[RESOLVED] : NIFI : LISTENHTTP SSL

[#1]

Hi all,

I'm trying use listenhttp with ssl.

I've read this post : http://www.simonellistonball.com/technology/nifi-ssl-listenhttp/

But i don't know how use keystore from client. It is the same keystore than keystore used in nifi.properties ?

thanks

[#2]

Hi @mayki wogno,

If you're talking about a situation where you've got

nifi.security.needClientAuth=True

in your nifi.properties, then for the client, you probably want to generate a separate cert that has been signed by the same CA that your NiFi node(s) trust. I.e. either use the same self-managed CA (such as the one that Ambari installs for you) or use official (mostly paid) CA-signed certificates.

As a proof of concept though, it's certainly fine to re-use the same certs.

Your private key could well be in pkcs12 with a passphrase protecting it, but you should be able to convert it to an unprotected pem file using something like this.

[root@test-nifti conf] # openssl pkcs12 -in keystore.pkcs12 -out keystore.pem -nodes

Then you can test connecting to itself using curl like so.

[root@test-nifti conf]# curl -v --cacert ./keys_test-nifti/nifi-cert.pem --cert ./keystore.pem https://test-nifti.example.com:9443/nifi
* About to connect() to test-nifti.example.com port 9443 (#0)
*   Trying 192.26.217.14...
* Connected to test-nifti.example.com (192.26.217.14) port 9443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: ./keys_test-nifti/nifi-cert.pem
  CApath: none
* NSS: client certificate from file
*       subject: CN=test-nifti.example.com,OU=EXAMPLE.COM
*       start date: Oct 17 23:14:07 2016 GMT
*       expire date: Oct 17 23:14:07 2019 GMT
*       common name: test-nifti.example.com
*       issuer: CN=nifti-ca.example.com,OU=EXAMPLE.COM
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA256
* Server certificate:
*       subject: CN=test-nifti.example.com,OU=EXAMPLE.COM
*       start date: Oct 17 23:14:07 2016 GMT
*       expire date: Oct 17 23:14:07 2019 GMT
*       common name: test-nifti.example.com
*       issuer: CN=nifti-ca.example.com,OU=EXAMPLE.COM
> GET /nifi HTTP/1.1
> User-Agent: curl/7.29.0
> Host: test-nifti.example.com:9443
> Accept: */*
> 
< HTTP/1.1 302 Found
< Date: Wed, 19 Oct 2016 12:33:50 GMT
< Location: https://test-nifti.example.com:9443/nifi/
< Content-Length: 0
< Server: Jetty(9.3.9.v20160517)
< 
* Connection #0 to host test-nifti.example.com left intact
[root@test-nifti conf]#

You can also import that keystore into your browser, so that the NiFi node can authenticate you.

The point of using a separate certificate though, is that you are not sharing the NiFi node(s) “secret” to your clients but instead using the mutual trust of the CA. So make sure you’re using dedicated client certs if you’re doing anything more than basic testing.

Please upvote and accept if this answered your question :).

[#3]

@Nik Lam : Need some details

In my cluster with 4 nodes, I've used nifi-toolkit for generating keystore/truststore + kerberos for accessing WEB UI. It is OK

In your example, where is come your file keystore?

keystore.pkcs12 

[#4]

It was generated using nifi-toolkit as well with "-T pkcs12".

[#5]

So you created .pkcs12 keystores instead of JKS keystores.

That is fine.... try using the below command to change passwords:

keytool -importkeystore -srckeystore keystore.pkcs12 -srcstoretype PKCS12 -srcstorepass 123456 -destkeystore keystore-new.pkcs12 -deststoretype PKCS12 -deststorepass 11223344 

or the following to change it to jks with new password:

keytool -importkeystore -srckeystore <keystore.pkcs12> -srcstoretype pkcs12
 -srcalias <alias> -destkeystore <keystore.jks>
 -deststoretype jks -deststorepass <new-password> -destalias <alias>

Thanks,

Matt

[#6]

@Nik Lam : Oh sorry, i don't speak about connecting to web ui

I would use listenhttp with SSL, so in ListenHTTP (Properties ==> use : SSL Context Service)

On the Processor Group Configuration, I need use SSL so how keystore/truststore need to use?

[#7 · Accepted solution]

@mayki wogno

SSL Certificates are generally issued per server. It would be extremely unusual to have multiple certificates issued for the same server or to issue certificates per piece of software. I am not saying you can't, but unnecessary.

Having said that, when you installed NiFi via Ambari, The NIfI CA that was installed likely generated the certificates needed for all your servers where NiFi is running its nodes. A quick look in the nifi.properties file for these jks files will reveal where to find them and the passwords for them as well (Default nifi.properties location on each node --> /etc/nifi/2.0.0.0-579/0/nifi.properties)

Now here is the hitch.... When the NiFi CA generates these keystores for your NiFi nodes, the keystore and truststore on every node end up with its own unique password. This will not work for the ssl context service you need to configure to make your ListenHTTP processor operate using SSL. This is because the same identical ssl context service will be used by the listenHTTP processor on each node.

There is a little work that needs to be done on each node to get you setup here, I suggest you make a copy of the keystore.jks and truststore.jks files on every node in to some new directory. Then capture the following lines from the nifi.properties file on each node:

nifi.security.keyPasswd=<keypass> 
nifi.security.keystore=/<path>/keystore.jks 
nifi.security.keystorePasswd=<keystore-password> 
nifi.security.keystoreType=jks 
nifi.security.truststore=/<path>/truststore.jks 
nifi.security.truststorePasswd=<truststore-password>
nifi.security.truststoreType=jks

Lets assume you create a /opt/nifi-certs/ directory n every node and placed them there.

Next you will want to use the keytool command to change the password on the copies so that nifi.security.keyPasswd= uses the same password on every node, nifi.security.keystorePasswd= uses the same password on every node, and nifi.security.truststorePasswd= uses the same password on every node. While the contents of each certificate is unique to the server, the passwords used to access that key and keystore will then all be the same. This will allow you to configure your ssl-context service to use common passwords to access these server keystores on each node.

Below are the command you can use to change the keystore/trustore passwords and the key passwords within the keystore files.

For each keystore do the following:

keytool -storepasswd -new <new_storepass> -keystore keystore.jks  

(Make sure you use the same <new_storepass> on every node in your nifi cluster.

keytool -keypasswd  -alias [Alias name for private key]  -keystore [path to key store]

(If you don't know your private key <alias>, you can get it by running the following command against your keystore.jks file. You will be prompted for yoru <new_storepassword> and the original <key-password>, and then allowed to set a >new-key-password>)

keytool -v --list -keystore keystore.jks

For each Truststore do the following:

keytool -storepasswd -new <new_trustpass> -keystore truststore.jks  

(Make sure you use the same <new_trustpass> on every node in your nifi cluster.

After you have completed the above on every node you are ready to configure and start using the ssl-context service.

Thanks,

Matt

[#8]

@mclark

I've found that was wrong in my config. Before changing storepass for keystore.jks. I need to change keypasswd for private key first

So Step 1 :

keytool -keypasswd  -alias [Alias name for private key]  -keystore [path to key store]

Step 2 :

keytool -storepasswd -new <new_storepass> -keystore keystore.jks

Now It works

Thanks @mclark

[#9]

@mclark : Oh thanks, I appreciate it.

Just for clear, for each node, I'll copy theirs own keystore/truststore in specific directory and change their passwd with same passwd for all nodes

Need I change something in the client with curl command?

curl -X POST -H 'Content-Type: application/json' -H 'filename: gstat-flume-a1.log.10' -F file=@/var/opt/hosting/log/flume/flume-a1.log.10 http://nifi011:10001/contentListener

[#10]

Your client curl command would not be pointing at a secure https://nifi011:10001/contentListener end-point and will need its own certificate to present in this connection. You can use the toolkit to generate another pkcs12 keystore that your connecting client can use. That way it gets signed by the same CA.

here is some useful information on this topic:

http://callistaenterprise.se/blogg/teknik/2011/04/04/curl-mutual-authentication-and-web-services/