Unable to start NiFI after i go HTTPS.?

[#1]

Hi,

I am using the latest version of NiFi and was working with HTTP recently. i tried to go HTTPS by doing the steps below and my NiFi wont start..

keytool -genkey -alias localhost -keyalg RSA -keysize 1024 -dname "CN=nifis2s,OU=nestle,O=nestle,L=selfsigned,ST=stl,C=us" -keypass testpass-keystore localhost.keystore.jks -storepass testpass

keytool -export -alias localhost -keystore localhost.keystore.jks -rfc -file localhost.cert -storepass testpass

keytool -import -noprompt -alias localhost -file localhost.cert -keystore localhost.truststore.jks -storepass testpass

it generated keystore.jks and truststore.jks files. and i pointed them in nifi.config and changed it use https. also added entry in to authorizers.xml with initial user as "CN=nifis2s,OU=nestle,O=nestle,L=selfsigned,ST=stl,C=us"

but when i tried to start up nifi after changes , it wont start up after some time it is shutting down with this errors below.

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'x509AuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: javax.xml.bind.UnmarshalException

does CN=nifis2s,OU=nestle,O=nestle,L=selfsigned,ST=stl,C=us need to be a valid user on the machine.?

i tried with different combinations like CN=nifiuser,OU=test

is there anything that i am missing.??

[#2]

I would never suggested someone should remove the authorizers.xml file. Both the users.xml and authorizations.xml files are built from the configurations in the authorizers.xml.

Did you try providing the absolute paths to your keystore and trustsore jks files in your nifi.properties file?

[#3]

@Matt

i thought of that , but all other setting are in the same format and working..so didnt change.

but now changed those 2 locations (truststore,keystore) to F:\Tools\HDF-2.0.1.0\nifi\conf\nifidev.keystore.jks

getting this error below..

2016-11-17 13:45:12,224 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.reflect.InvocationTargetException java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_77] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_77] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_77] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_77] at org.apache.nifi.NiFi.<init>(NiFi.java:146) ~[nifi-runtime-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] at org.apache.nifi.NiFi.main(NiFi.java:243) ~[nifi-runtime-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] Caused by: java.lang.IllegalArgumentException: java.net.MalformedURLException: unknown protocol: f at org.eclipse.jetty.util.ssl.SslContextFactory.setKeyStorePath(SslContextFactory.java:561) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.server.JettyServer.configureSslContextFactory(JettyServer.java:618) ~[nifi-jetty-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] at org.apache.nifi.web.server.JettyServer.createSslContextFactory(JettyServer.java:603) ~[nifi-jetty-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] at org.apache.nifi.web.server.JettyServer.configureConnectors(JettyServer.java:587) ~[nifi-jetty-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] at org.apache.nifi.web.server.JettyServer.<init>(JettyServer.java:141) ~[nifi-jetty-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12] ...

6 common frames omitted

Caused by: java.net.MalformedURLException: unknown protocol: f at java.net.URL.<init>(URL.java:600) ~[na:1.8.0_77] at java.net.URL.<init>(URL.java:490) ~[na:1.8.0_77] at java.net.URL.<init>(URL.java:439) ~[na:1.8.0_77] at org.eclipse.jetty.util.resource.Resource.newResource(Resource.java:166) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.resource.Resource.newResource(Resource.java:149) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.ssl.SslContextFactory.setKeyStorePath(SslContextFactory.java:557) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] ... 10 common frames omitted Suppressed: java.io.IOException: The filename, directory name, or volume label syntax is incorrect at java.io.WinNTFileSystem.canonicalize0(Native Method) ~[na:1.8.0_77] at java.io.WinNTFileSystem.canonicalize(WinNTFileSystem.java:428) ~[na:1.8.0_77] at java.io.File.getCanonicalPath(File.java:618) ~[na:1.8.0_77] at java.io.File.getCanonicalFile(File.java:643) ~[na:1.8.0_77] at org.eclipse.jetty.util.resource.Resource.newResource(Resource.java:179) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] ... 12 common frames omitted

2016-11-17 13:45:12,226 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...

[#4]

In Windows you need to prepend the file path with "file:///". Try "file:///F:/Tools/HDF-2.0.1.0/nifi/conf/nifidev.keystore.jks".

[#5]

@Matt , @Andy LoPresto

i had a typo earlier when i used ./config/nifidev.keystore.jks...i found it and corrected it.so no need to go windows structure.

it did startup sucessfully..with the below trace..

but when i tried to go to https://10.242.27.40:9443/nifi or https://127.0.0.1:9443/nifi i am not able to access the site.

i thought it would auto login in with the admin user..do i have to do anything else.??

2016-11-17 14:25:07,302 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@7bf2e475{SSL,[ssl, http/1.1]}{0.0.0.0:9443}

2016-11-17 14:25:07,302 INFO [main] org.eclipse.jetty.server.Server Started @97001ms

2016-11-17 14:25:08,009 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:

2016-11-17 14:25:08,009 INFO [main] org.apache.nifi.web.server.JettyServer https://10.242.27.40:9443/nifi

2016-11-17 14:25:08,009 INFO [main] org.apache.nifi.web.server.JettyServer https://127.0.0.1:9443/nifi

2016-11-17 14:25:08,011 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap

Regards,

Sai

[#6]

Did you load your client certificate into the browser? What do you mean you are not able to access the site? Are you getting a TLS error and nothing comes up, or are you getting a NiFi message that says you are not authorized? You need to ensure that the DN of your client cert is in conf/authorizers.xml as the Initial Admin Identity and that it matches the user defined in conf/users.xml. You can find more information by tailing $NIFI_HOME/logs/nifi-app.log and $NIFI_HOME/logs/nifi-user.log.

[#7]

@Andy LoPresto

no , i did not load any certificates into browser. i didn't set the the property in nifi config nifi.security.needClientAuth=

also how can i generate a client certificate..?? i tried to upload the cert generated by the command below to the browser..this is the same one which i used to add to keystore and truststore.

keytool -genkey -alias nifidev -keyalg RSA -keysize 1024 -dname "CN=nifistos,OU=nestle" -keypass testpwd -keystore nifidev.keystore.jks -storepass testpwd

i have the Initial Admin Identity as CN=nifistos,OU=nestle and also i see a user for that in users.xml

so all i did was 1. generate a certificate using above command 2. add it to keystore and trust store

3 in nifi.config set https and keystore & truststore to generated files above 4.add initial admin to the dname that was used above.

i don't have any certificate authority and\or client certificates or anything else apart from those i did above..

please let me know if i am missing any.

Regards,

Sai

[#8]

You did not answer my question about what type of error you are actually encountering.

The setting nifi.security.needClientAuth applies only to cluster communications and has no bearing on UI/API access. The Admin Guide discusses this.

If you did not configure LDAP or Kerberos as a user authentication mechanism, then you will definitely need a client certificate in order to access the secured UI. If you manually generated your keystore and truststore, you will need to do the same with the client certificate and have it be signed by a key that is loaded into your truststore. You can also use the TLS Toolkit that is provided with NiFi to generate all of the necessary files. There are many step-by-step tutorials on this site and others. An easy one is here. Do not load the server certificate identifying the NiFi application into the browser to act as a client certificate. They are different, and exposing the private key of the NiFi certificate is not a good idea.

[#9]

@Andy LoPresto

I am not seeing any errors or warnings in app and user log files. Also I am not using LDAP or Kerberos as I am trying to test this locally. here is the error that I am getting (https://localhost:9443/nifi).

I generated keystore and truststore using keytool command shown in above posts.

since I have not generated any client certificates I will try to do that now.

is there a keytool command (or options) that can generate client certificate and have it be signed by a key that is loaded into the truststore..?

i tried to use this as mentioned in the link you shared here. but it is not generating the client certificates..

its generating keystore,truststore and nifi-cert etc expect client certs..

pls check the last line below for error.

I let Bryan know about this , he said it may be because of windows.

tls-toolkit.bat standalone -n 'localhost' -C 'CN=nifistos, OU=mycompany' -o './target'

2016-11-18 10:17:03,193 INFO [main] o.a.n.t.t.s.TlsToolkitStandaloneCommandLine No nifiPropertiesFile specified, using embedded one.

2016-11-18 10:17:03,316 INFO [main] o.a.n.t.t.s.TlsToolkitStandalone Running standalone certificate generation with output directory ..\bin

2016-11-18 10:17:04,142 INFO [main] o.a.n.t.t.s.TlsToolkitStandalone Generated new CA certificate ..\bin\nifi-cert.pem and key ..\bin\nifi-key.key

2016-11-18 10:17:04,147 INFO [main] o.a.n.t.t.s.TlsToolkitStandalone No hostnames specified, not generating any host certificates or configuration.

2016-11-18 10:17:04,153 INFO [main] o.a.n.t.t.s.TlsToolkitStandalone Writing new ssl configuration to ..\bin\'localhost'

2016-11-18 10:17:04,426 INFO [main] o.a.n.t.t.s.TlsToolkitStandalone Successfully generated TLS configuration for 'localhost' 1 in ..\bin\'localhost'

Error creating generating tls configuration. (Unknown object id - 'CN - passed to distinguished name)

[#10]

You may be able to run the TLS Toolkit command on Windows as follows:

tls-toolkit.bat ""standalone" -n "localhost" -C "CN=nifistos, OU=mycompany" -o "./target""

[#11]

@Andy LoPresto

i am still not able to solve this , i did try all the things from all posts. i am trying all this on my local laptop. could there be any issue in my network\firewall settings that is preventing me from accessing the UI.

generated a certificate for server, imported it into truststore..generated a client certificate imported into the same truststore.

point keystore and truststore in nifi.config. change the https port . i did set initial admin as the client DN .the NiFi service is running but i am not able to get to the UI. its not asking to to select a certificate or import one. i did anyway..

but still getting the same error as above screen..no errors are reported in app.log or user.log

anything else that can cause this..??

Regards,Sai