I'm needing to clean a NiFi Node to have it stop getting errors. I've narrowed down the problem quite a bit (I'm trying to have it Authenticate via LDAP, with a self-signed SSL cert, both of these things are set up properly now).
However, because I've been through so much troubleshooting and trial and error with this NiFi server, I have experienced WAY more hassle than I would've expected. I've been trying to get this to work for nearly a month now.
SSL certs, truststore, keystore, node config (In CM) are all set up to the best of my ability. I stop the service roles in CM, put host in maintenance mode, restart the VM nifi is running on (to clear extra scm-agent processes).
I've moved the flowfile.xml.gz, deleted users.xml, authorizations.xml, cleared the _repository folders out (moving them to a backup folder), cleared the archive folder, state/local folder.
I've set initial admin to our proper LDAP identity. (I never get to this stage before the node shuts off now though)
Jetty is reporting that the there is no valid keystore, but I am not sure that this is the cause of the effect of a different problem. I've been very careful to create the keystores exactly to specification following @MattWho 's article and have verified everything, also I had HTTPS working last night (csr worked, but I could not manage to log in due to "unverified keystore" on the client side I believe).
Having as little experience with this sort of thing as I do, I have been so challenged and puzzled to get HTTPS set up for LDAP auth. I keep feeling like I'm only one or two steps away from having things working but then another problem springs up.
Here is a piece of the log file from which I've scoured and this is the FIRST sign of something not going correctly:
| 7:04:23.327 PM | INFO | _nifi | No Spring WebApplicationInitializer types detected on classpath |
| 7:04:23.404 PM | INFO | ContextHandler | Started o.e.j.w.WebAppContext@5b16e486{nifi,/nifi,file:///var/lib/nifi/work/jetty/nifi-web-ui-1.9.0.1.0.1.0-12.war/webapp/,AVAILABLE}{/var/lib/nifi/work/nar/framework/nifi-framework-nar-1.9.0.1.0.1.0-12.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-ui-1.9.0.1.0.1.0-12.war} |
| 7:04:23.719 PM | INFO | AnnotationConfiguration | Scanning elapsed time=181ms |
| 7:04:23.748 PM | INFO | _nifi_api | No Spring WebApplicationInitializer types detected on classpath |
Followed by this Error, which is followed by a MASSIVE list of missing beans which i have not included.
Context initialization failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'jwtService' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtService' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'keyService' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyService' defined in class path resource [nifi-administration-context.xml]: Cannot resolve reference to bean 'keyTransactionBuilder' while setting bean property 'transactionBuilder'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyTransactionBuilder' defined in class path resource [nifi-administration-context.xml]: Cannot resolve reference to bean 'keyDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyDataSource': FactoryBean threw exception on object creation; nested exception is org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /var/lib/nifi/database_repository/nifi-user-keys.lock.db" [8000-176]
Warn:
Error opening database: "Could not save properties /var/lib/nifi/database_repository/nifi-user-keys.lock.db" [8000-176]
Failed startup of context o.e.j.w.WebAppContext@2b85edc7{nifi-api,/nifi-api,file:///var/lib/nifi/work/jetty/nifi-web-api-1.9.0.1.0.1.0-12.war/webapp/,UNAVAILABLE}{/var/lib/nifi/work/nar/framework/nifi-framework-nar-1.9.0.1.0.1.0-12.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.9.0.1.0.1.0-12.war} And the final error before shutting down:
| 7:04:33.701 PM | INFO | _ | No Spring WebApplicationInitializer types detected on classpath |
| 7:04:33.755 PM | INFO | ContextHandler | Started o.e.j.w.WebAppContext@490704a5{nifi-error,/,file:///var/lib/nifi/work/jetty/nifi-web-error-1.9.0.1.0.1.0-12.war/webapp/,AVAILABLE}{/var/lib/nifi/work/nar/framework/nifi-framework-nar-1.9.0.1.0.1.0-12.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.0.1.0.1.0-12.war} |
| 7:04:33.787 PM | WARN | JettyServer | Failed to start web server... shutting down.
java.lang.IllegalStateException: no valid keystore
at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:50)
at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1071)
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:262)
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:229)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:72)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:279)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:398)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297) |
| 7:04:33.797 PM | INFO | NiFi | Initiating shutdown of Jetty web server... |
So. With this, it is failing to even reach the stage where it will generate the users.xml and authorizations.xml file that I'm used to it auto-generating after I remove it. I've had to change the initial admin a couple of times, so I learned how to do that without reinstalling.
I've removed the flowfile.xml.gz, which I suspect may be an issue here. As well as the _repository folders, which I was advised to do via other forum posts here, but I may have cleared more than I should have? I have backups saved. Really my big question is: what all CAN i remove/clean without NiFi not being able to recover / auto-gen new files on start.
Any ideas would be super appreciated!
