JWT token validation error but only single-user login is enabled

[#1]

Hi, very weird issue here: I have a three-node NiFi 1.16.3 cluster that I deployed several weeks ago in http-mode with all security options disabled for the sake of simplicity since I'm totally new to NiFi. Hence, no login mechanism whatsoever.

I've configured several flows and the whole thing has been working flawlessly for weeks.

Then I decided to activate all the security options and single-user login. To do so without impacting the operations of the existing cluster, I worked on a new one. Created new keystores and truststores with NiFi tls-toolkit, changed nifi.properties accordingly, created the user on each node with /nifi.sh set-single-user-credentials USER PASSWORD, etc. Once everything was working in the new cluster, I turned the configurations into an Ansible playbook, destroyed the new Nifi cluster and applied the playbook on a new one, created from scratch, several times to make sure the configuration was rock-solid. 

After all this, I applied the same Ansible playbook to the existing cluster to enable security and single-user login:

nifi.security.user.authorizer=single-user-authorizer

nifi.security.user.login.identity.provider=single-user-provider

Upon restart, the old cluster went up with no issues and all the flows kept working as usual. 

But I have been able to login to the cluster's web UI only once after the restart of the NiFi service, then for apparently no reason the web interface became accessible only if I enable anonymous mode

nifi.security.allow.anonymous.authentication=true

Every time I try to login, all I get is a JWT token validation error from org.springframework.security.oauth2.server.resource.InvalidBearerTokenException, even if I am not using Oauth2 at all: 

2022-08-01 09:29:31,004 ERROR [NiFi Web Server-511] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: org.springframework.security.oauth2.server.resource.InvalidBearerTokenException: An error occurred while attempting to decode the Jwt: Signed JWT rejected: Another algorithm expected, or no matching key(s) found. Returning Internal Server Error response.

org.springframework.security.oauth2.server.resource.InvalidBearerTokenException: An error occurred while attempting to decode the Jwt: Signed JWT rejected: Another algorithm expected, or no matching key(s) found

 at org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationProvider.getJwt(JwtAuthenticationProvider.java:101)

 at org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationProvider.authenticate(JwtAuthenticationProvider.java:88)

 at org.apache.nifi.web.api.AccessResource.getAccessStatus(AccessResource.java:252)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 at java.base/java.lang.reflect.Method.invoke(Method.java:566)

 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)

 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)

 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)

 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)

 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)

 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)

 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)

 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)

 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)

 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)

 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)

 at org.glassfish.jersey.internal.Errors.process(Errors.java:292)

 at org.glassfish.jersey.internal.Errors.process(Errors.java:274)

 at org.glassfish.jersey.internal.Errors.process(Errors.java:244)

 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)

 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)

 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)

 at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)

 at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)

 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)

 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)

 at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)

 at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)

 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)

 at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)

 at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:204)

 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)

 at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)

 at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487)

 at org.apache.nifi.web.server.filter.DataTransferExcludedDoSFilter.doFilterChain(DataTransferExcludedDoSFilter.java:51)

 at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336)

 at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.apache.nifi.web.server.log.RequestAuthenticationFilter.doFilterInternal(RequestAuthenticationFilter.java:59)

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.apache.nifi.web.security.headers.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:48)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)

 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)

 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)

 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)

 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)

 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)

 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)

 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)

 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)

 at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)

 at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)

 at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)

 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

 at org.eclipse.jetty.server.Server.handle(Server.java:516)

 at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)

 at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)

 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)

 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)

 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

 at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)

 at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)

 at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)

 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

 at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)

 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)

 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)

 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

 at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: org.springframework.security.oauth2.jwt.BadJwtException: An error occurred while attempting to decode the Jwt: Signed JWT rejected: Another algorithm expected, or no matching key(s) found

 at org.springframework.security.oauth2.jwt.NimbusJwtDecoder.createJwt(NimbusJwtDecoder.java:180)

 at org.springframework.security.oauth2.jwt.NimbusJwtDecoder.decode(NimbusJwtDecoder.java:137)

 at org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationProvider.getJwt(JwtAuthenticationProvider.java:97)

 ... 104 common frames omitted

Caused by: com.nimbusds.jose.proc.BadJOSEException: Signed JWT rejected: Another algorithm expected, or no matching key(s) found

 at com.nimbusds.jwt.proc.DefaultJWTProcessor.process(DefaultJWTProcessor.java:357)

 at com.nimbusds.jwt.proc.DefaultJWTProcessor.process(DefaultJWTProcessor.java:303)

 at org.springframework.security.oauth2.jwt.NimbusJwtDecoder.createJwt(NimbusJwtDecoder.java:154)

 ... 106 common frames omitted

I want to stress again that the deployment and configuration of the NiFi cluster is entirely managed by an Ansible playbook that I've tested thoroughly several times on identical clusters created from scratch, and the login procedure has been working flawlessly on them. The issue appears only on an already existing cluster that has been working with the security features disabled for several weeks.

Also, I've already deleted cookies and tried to login in incognito mode with both Chrome and Firefox.

Could someone please point me in the right direction? This is driving me crazy. Thank you all.

W