Gold ⭐️

Solved

NiFi UI is Unstable when it connected with cluster nodes

8 replies
3985 views
Translate All

Hi Team,

we have an issue with Nifi UI, which is unstable with multiple nodes (like 401 unauthorized error) but with single node it is working fine without any issue, For multiple nodes also UI coming but dropping off after sometime, and accesible to only to few users randomly
coming to the background of this issue, we recently migrated our application from DHC CASS to DYP CAAS (AWS), all configurations are fine and backend also working fine. only thing changed is the domain name (hostname changed) we replaced this latest hostname in all config files (users,nifi.properties, autherizations.xml, sts, deployment) wherever it is needed in place of Old domain name.

from Browser network logs what we observed is 401 unauthorized error whenever its throwing error logs unauthorized Please check logs.
and these errors everytime changing to different api's in the nifi not consistent, sometimes its coming with config, sometimes summary, sometimes current user
ex:

equest URL

https://nifi.intra.net/nifi-api/flow/process-groups/root?uiOnly=true

Request Method

GET

Status Code

401 Unauthorized

when we checked the error :

www-authenticate

Bearer error="invalid_token", error_description="An error occurred while attempting to decode the Jwt: Signed JWT rejected: Another algorithm expected, or no matching key(s) found"
FYI, we are using OIDC properties before we login to the Nifi UI, the SSO login successful but UI is unstable somehow with above error i mentioned

from pod logs observed like:
2025-07-01 09:45:05,637 ERROR [NiFi Web Server-3014] o.apache.nifi.web.api.OIDCAccessResource OIDC Request [d7a8a7ae-8c13-43d4-86ef-ba48c9e0effb] State [pos5kmbng7u6tsi2jca5v41019] not valid
2025-07-01 09:45:05,785 ERROR [NiFi Web Server-3014] o.g.j.server.ServerRuntime$Responder Error while closing the output stream in order to commit response.
java.lang.IllegalStateException: WRITER

And also from user logs observed like:
2025-07-07 14:55:18,873 INFO [NiFi Web Server-20941] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 192.168.246.30 [<PRG><CN=nifi.intra.net, OU=IT/FI, O=My Group AG, L=Stutt, ST=BI, C=BE><CN=nifi.intra.net, OU=IT/FI, O=My Group AG, L=Stutt, ST=BI, C=BE>] GET https://ss-nifi-1.svc-nifi-cluster.nifi-test.svc.cluster.local:8443/nifi-api/flow/controller/bulletins
2025-07-07 14:55:18,873 INFO [NiFi Web Server-20941] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [PRG] 192.168.246.30 GET https://ss-nifi-1.svc-nifi-cluster.nifi-test.svc.cluster.local:8443/nifi-api/flow/controller/bulletins
2025-07-07 14:57:35,841 WARN [NiFi Web Server-20941] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Committed. Returning Conflict response.
java.lang.IllegalStateException: Committed

we already checked in OIDC JWT as well by decrypting the JWT token that we have in the browser , they told kid is not matching in their OIDC , So can you please help us to sort out this issue. where this JWT token rejection happening.

@MattWho

@SAMSAL

Thanks

Apache NiFi

PradNiFi1236Answer

Gold ⭐️

thanks @MattWho for replying we were able to sort it out after we enabled sticky sessions on load balancer target group attributes.

MattWho

Titanium ⭐️⭐️⭐️

@PradNiFi1236

Not much here to work with. I suggest first comparing the NiFi configuration files between all your nodes to make sure they are all are identical with exception of hostnames, keystores, and truststores.

Are you using a load balancer? If so, do you see a change in behavior if you enable session affinity (Sticky sessions) in your LB?

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

PradNiFi1236Author

Gold ⭐️

Thanks for the reply @MattWho , yes we do compared all the config files are same in both nodes including truststore and keystore , we do see previous certs in keystore and truststore do we need to delete those as well? because in existing on premises even though we have old certs in truststore and keystore , that cluster running fine.

we do have loadblancer , yes we are maintaining session affinity but no luck with that too.
selector:
app: nifi
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800

Translate

MattWho

Titanium ⭐️⭐️⭐️

@PradNiFi1236

The keystore the NiFi node uses must contain only 1 clientAuth private key. There is no configuration option in Apache NiFi to specific which private key is used when multiple exist in the keystore. Of course the truststore will be same on all the nodes and can contain as many Public keys as you want, but must contain the complete trust chain fro yoru NiFi node private certificates.

Translate

PradNiFi1236Author

Gold ⭐️

yes @MattWho , we are maintaining single privatekey only even after that we are having same issue

Translate

PradNiFi1236Author

Gold ⭐️

as i mentioned earlier this is the error from pod logs

2025-07-10 08:42:12,535 ERROR [NiFi Web Server-23433] o.apache.nifi.web.api.OIDCAccessResource OIDC Request [03c8123c-11df-416c-8afa-40cab2055d57] State [bokp6imokp4thp7ue74p44das3] not valid
2025-07-10 08:42:12,630 ERROR [NiFi Web Server-23433] o.g.j.server.ServerRuntime$Responder Error while closing the output stream in order to commit response.
java.lang.IllegalStateException: WRITER
at org.eclipse.jetty.server.Response.getOutputStream(Response.java:776)
at javax.servlet.ServletResponseWrapper.getOutputStream(ServletResponseWrapper.java:142)
at org.glassfish.jersey.servlet.internal.ResponseWriter.writeResponseStatusAndHeaders(ResponseWriter.java:156)
at org.glassfish.jersey.server.ServerRuntime$Responder$1.getOutputStream(ServerRuntime.java:625)
at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:171)
at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:276)
at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232)
at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247)
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:241)
at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:707)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:419)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:263)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:204)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:748)

Translate

PradNiFi1236Author

Gold ⭐️

@MattWho , Could you please reply on it, this is related to Session affinity , even though we are maintaining at LoadBalancer its still throwing error on multinode

Unauthorized

log out
home

Unauthorized error="invalid_token", error_description="An error occurred while attempting to decode the Jwt: Expired JWT", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"

Translate

MattWho

Titanium ⭐️⭐️⭐️

@PradNiFi1236

Really not sure why NiFi is receiving an Expired Token from your load balancer. This would require some multiple service deep dive to work through which is not something I can do through this community forum. This is where using Cloudera products and having a Cloudera license would enable you to take advantage of the Cloudera Professional Services that can deep dive and dive it to setting up workable environments.

Translate

PradNiFi1236AuthorAnswer

Gold ⭐️

thanks @MattWho for replying we were able to sort it out after we enabled sticky sessions on load balancer target group attributes.

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.