NiFi-1.14.0 failure to start: Unable to start Flow Controller.

[#1]

Hey Folks, I'm unable to get 1.14.0 to run on my linux box, it appears to be unhappy with configuring SSL services.  Any advice or suggestions are welcome. 

2021-08-03 18:54:06,172 WARN [main] o.a.n.d.html.HtmlDocumentationWriter Could not link to org.apache.nifi.ssl.RestrictedSSLContextService because no bundles were found for ListenFTP
2021-08-03 18:54:06,172 WARN [main] o.a.n.d.html.HtmlDocumentationWriter Could not link to org.apache.nifi.ssl.SSLContextService because no bundles were found for ListenFTP
2021-08-03 18:54:07,225 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,225 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,225 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,225 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,231 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,234 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,241 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,241 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,241 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,241 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,247 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,247 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,248 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,248 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,248 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size
2021-08-03 18:54:07,250 WARN [main] net.schmizz.sshj.DefaultConfig Illegal key size or default parameters
2021-08-03 18:54:07,251 WARN [main] net.schmizz.sshj.DefaultConfig Disabling high-strength ciphers: cipher strengths apparently limited by JCE policy
2021-08-03 18:54:09,670 WARN [main] o.a.n.d.html.HtmlDocumentationWriter Could not link to org.apache.nifi.ssl.SSLContextService because no bundles were found for HandleHttpResponse
2021-08-03 18:54:11,445 INFO [main] org.eclipse.jetty.server.Server jetty-9.4.42.v20210604; built: 2021-06-04T17:33:38.939Z; git: 5cd5e6d2375eeab146813b0de9f19eda6ab6e6cb; jvm 1.8.0_73-b02
2021-08-03 18:54:12,203 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=355ms
2021-08-03 18:54:12,294 INFO [main] org.eclipse.jetty.server.session DefaultSessionIdManager workerName=node0
2021-08-03 18:54:12,294 INFO [main] org.eclipse.jetty.server.session No SessionScavenger set, using defaults
2021-08-03 18:54:12,296 INFO [main] org.eclipse.jetty.server.session node0 Scavenging every 600000ms
2021-08-03 18:54:12,344 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@7336fd8f{/nifi-update-attribute-ui-1.14.0,/nifi-update-attribute-ui-1.14.0,file:///u/users/<username>/nifi-1.14.0/work/jetty/nifi-update-attribute-ui-1.14.0.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-update-attribute-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-update-attribute-ui-1.14.0.war}
2021-08-03 18:54:12,651 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=145ms
2021-08-03 18:54:12,688 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@701c223a{nifi-standard-content-viewer,/nifi-standard-content-viewer-1.14.0,file:///u/users/<username>/nifi-1.14.0/work/jetty/nifi-standard-content-viewer-1.14.0.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-standard-content-viewer-1.14.0.war}
2021-08-03 18:54:13,279 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=255ms
2021-08-03 18:54:13,316 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@550c973e{/nifi-jolt-transform-json-ui-1.14.0,/nifi-jolt-transform-json-ui-1.14.0,file:///u/users/<username>/nifi-1.14.0/work/jetty/nifi-jolt-transform-json-ui-1.14.0.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-jolt-transform-json-ui-1.14.0.war}
2021-08-03 18:54:13,627 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=56ms
2021-08-03 18:54:13,633 INFO [main] o.e.j.s.handler.ContextHandler._nifi No Spring WebApplicationInitializer types detected on classpath
2021-08-03 18:54:13,664 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@161d95c6{nifi,/nifi,file:///u/users/<username>/nifi-1.14.0/work/jetty/nifi-web-ui-1.14.0.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-server-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-ui-1.14.0.war}
2021-08-03 18:54:13,958 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=116ms
2021-08-03 18:54:13,959 INFO [main] o.e.j.s.handler.ContextHandler._nifi_api No Spring WebApplicationInitializer types detected on classpath
2021-08-03 18:54:14,006 INFO [main] o.e.j.s.handler.ContextHandler._nifi_api Initializing Spring root WebApplicationContext
2021-08-03 18:54:15,879 INFO [main] o.a.n.p.AbstractBootstrapPropertiesLoader Determined default application properties path to be '/u/users/<username>/nifi-1.14.0/./conf/nifi.properties'
2021-08-03 18:54:15,880 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 202 properties from /u/users/<username>/nifi-1.14.0/./conf/nifi.properties
2021-08-03 18:54:16,593 WARN [main] org.apache.nifi.util.FlowParser Flow Configuration does not exist or was empty
2021-08-03 18:54:18,067 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE.  Building now...
2021-08-03 18:54:18,108 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 107 properties from system properties and environment variables
2021-08-03 18:54:18,108 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 107 properties.  Including precedence overrides effective accessible registry key size is 107
2021-08-03 18:54:18,136 WARN [main] o.a.nifi.security.util.SslContextFactory Some truststore properties are populated (./conf/truststore.p12, null, PKCS12) but not valid
2021-08-03 18:54:18,136 ERROR [main] o.apache.nifi.controller.FlowController Unable to start the flow controller because the TLS configuration was invalid: The truststore properties are not valid
2021-08-03 18:54:18,138 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@5203c80f{nifi-api,/nifi-api,file:///u/users/<username>/nifi-1.14.0/work/jetty/nifi-web-api-1.14.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.14.0.war}
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
	at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:83)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1129)
	at org.apache.nifi.NiFi.<init>(NiFi.java:159)
	at org.apache.nifi.NiFi.<init>(NiFi.java:71)
	at org.apache.nifi.NiFi.main(NiFi.java:303)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: Flow controller TLS configuration is invalid
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:176)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:101)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1884)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getObjectForBeanInstance(AbstractAutowireCapableBeanFactory.java:1266)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:345)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1160)
	at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:51)
	... 38 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: Flow controller TLS configuration is invalid
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:176)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:101)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1884)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getObjectForBeanInstance(AbstractAutowireCapableBeanFactory.java:1266)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:345)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1160)
	at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169)
	... 45 common frames omitted
Caused by: java.lang.IllegalStateException: Flow controller TLS configuration is invalid
	at org.apache.nifi.controller.FlowController.<init>(FlowController.java:485)
	at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:392)
	at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:83)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:169)
	... 53 common frames omitted
Caused by: org.apache.nifi.security.util.TlsException: The truststore properties are not valid
	at org.apache.nifi.security.util.SslContextFactory.getTrustManagers(SslContextFactory.java:181)
	at org.apache.nifi.security.util.SslContextFactory.createSslContext(SslContextFactory.java:66)
	at org.apache.nifi.controller.FlowController.<init>(FlowController.java:482)
	... 56 common frames omitted
2021-08-03 18:54:18,506 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=126ms
2021-08-03 18:54:18,531 INFO [main] o.e.j.s.h.C._nifi_content_viewer No Spring WebApplicationInitializer types detected on classpath
2021-08-03 18:54:18,536 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@439f2d87{nifi-content-viewer,/nifi-content-viewer,file:///u/users/<username>/nifi-1.14.0/work/jetty/nifi-web-content-viewer-1.14.0.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-server-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-content-viewer-1.14.0.war}
2021-08-03 18:54:18,572 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=13ms
2021-08-03 18:54:18,573 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default
2021-08-03 18:54:18,573 INFO [main] o.e.j.s.h.ContextHandler._nifi_docs No Spring WebApplicationInitializer types detected on classpath
2021-08-03 18:54:18,614 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@721bf7ad{nifi-docs,/nifi-docs,file:///u/users/<username>/nifi-1.14.0/work/jetty/nifi-web-docs-1.14.0.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-server-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-docs-1.14.0.war}
2021-08-03 18:54:18,652 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=15ms
2021-08-03 18:54:18,654 INFO [main] o.e.j.server.handler.ContextHandler._ No Spring WebApplicationInitializer types detected on classpath
2021-08-03 18:54:18,682 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@852ef8d{nifi-error,/,file:///u/users/<username>/nifi-1.14.0/work/jetty/nifi-web-error-1.14.0.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-server-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.14.0.war}
2021-08-03 18:54:18,715 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.lang.IllegalStateException: /u/users/<username>/nifi-1.14.0/conf/keystore.p12 is not a valid keystore
	at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:50)
	at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
	at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
	at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.server.Server.doStart(Server.java:401)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1129)
	at org.apache.nifi.NiFi.<init>(NiFi.java:159)
	at org.apache.nifi.NiFi.<init>(NiFi.java:71)
	at org.apache.nifi.NiFi.main(NiFi.java:303)
2021-08-03 18:54:18,723 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2021-08-03 18:54:18,727 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@426c0486{SSL, (ssl, http/1.1)}{127.0.0.1:8443}
2021-08-03 18:54:18,727 INFO [Thread-1] org.eclipse.jetty.server.session node0 Stopped scavenging

[#2]

Looking at the version, it seems you are using an unsupported version. (The 1.14 version is supported in some forms but not currently for the one that can be deployed on an arbitrary box).

Most time people have problems with installation of software like NiFi the problem can be solved by using one of the supported versions, which can be found here: https://www.cloudera.com/downloads

For NiFi specifically look under CDF (Cloudera Data Flow).

[#3]

@zrbear 

It would  be difficult to assist here without you sharing the following:
- contents of nifi.properties file
- verbose listing of yoru keystore and truststore using keytool 

Also make sure if using Java JDK 8 that you have installed the unlimited strength JCE policies:
https://www.oracle.com/java/technologies/javase-jce8-downloads.html

Hope this helps,

Matt

[#4]

Hi,

The problem is apparently related to the certificates that are included, because if you generate them using nifi-toolkit, this problem is solved.

Here are the steps that I did:

cd / opt / installers
wget https://www.apache.org/dyn/closer.lua?path=/nifi/1.14.0/nifi-toolkit-1.14.0-bin.tar.gz
tar -xvfz nifi-toolkit-1.14.0-bin.tar.gz
mv nifi-toolkit-1.14.0 ../nifi-toolkit-1.14.0
cd ../nifi-toolkit-1.14.0/bin
./tls-toolkit.sh standalone -n "<IP_NIFI_SERVER>"
cd IP_NIFI_SERVER
cp * $NIFI_HOME / conf

<Start Nifi>

[#5]

Late response but... Are you running in FIPS mode?