Securing S2S between two secured NiFi instances on two cloud computers

[#1]

Hi,

I am trying to set up two NiFis (v1.4.0) on separate AWS EC2 instances and have them communicate securely.

I have secured each NiFi so you need SSL authentication to access the UI on a browser.

However I am not able to have the two instances communicate S2S.

I have had success with UNSECURED NiFi instances with them communicating on RAW protocol. However I don't think it's secure so would like to lock down the S2S comms, as well as NiFi UI access as well.

I'm pretty sure the properties nifi.remote.input.* and nifi.web.* are the ones to modify, but am confused as I can't seem find an example on the web for securing both the UI browser access and S2S comms.

Could somebody please point me in the right direction?

Thank you

[#2]

Bryan Bende has written a good tutorial here. Do you have a specific error message or failure you can share to help diagnose your issue?

[#3]

Hi Andy thanks for the response, I'll try to adapt the Bryan's tutorial to my application and report back.

His tutorial seems to be for two NiFi instances running on one machine. My application is two NiFis running on separate cloud machines. Not sure if it makes a big difference.

In the meantime here's the errors I get on my RPG. It's interesting that it seems to be referencing the 5... IP address. That was used on an EC2 instance that I created an image from, and I created this instance from that image.

Thank you

[#4]

The error you are getting indicates that the server certificate presented by the other NiFi instance has a Subject Alternative Name that is different from the expected hostname. You'll need to add the hostname into the list of SANs. This is a newer policy described by RFC 6125.

[#5]

Hi I have just had a go at the tutorial but there's a few aspects that is confusing me.

The toolkit will generate nifi.property files for the two instances but their S2S properties are as follows:

# Site to Site properties instance 1

nifi.remote.input.host=localhost

nifi.remote.input.secure=true

nifi.remote.input.socket.port=10443

nifi.remote.input.http.enabled=true

ifi.remote.input.http.transaction.ttl=30 sec

# Site to Site properties instance 2

nifi.remote.input.host=localhost

nifi.remote.input.secure=true

nifi.remote.input.socket.port=10444

nifi.remote.input.http.enabled=true

nifi.remote.input.http.transaction.ttl=30 sec

Would I be correct in saying if I set up an RPG on instance 2 to point to Instance 1, I'd have to set it up with https://<Instance 1 IP>:10443/nifi ?

How does the instance know which user to use, to access the ports etc? if I set up more users with less privileges, how would the instance know which one to use?

I am guessing that because the nifi.remote.input.secure is set to true, it is not possible to secure a RAW protocol?

Thank you

[#6]

For Site to Site, you configure the remote process group on the "originating"/"source" NiFi to point to the URL of the "remote"/"destination" NiFi and it will negotiate the rest on its own. So in your RPG, you would provide the URL "https://<instance 1 IP>:8443/nifi" assuming 8443 is the port that the NiFi UI/API is running on. You can also secure RAW transport connections as well -- it simply uses a secure socket connection.

There is more information contained in the following resources:

* Apache NiFi Admin Guide - Site to Site Properties

* Apache NiFi User Guide - Site to Site

* Apache NiFi Feature Description -- HTTP(S) for Site to Site (documents existing RAW capability as well)

[#7]

Because the error message was referencing a previous instance IP, I have re-done the keystore and truststore on both instances using the TLS Toolkit.

The commands I used on each instance were

/data/download/nifi-toolkit-1.4.0/bin/tls-toolkit.sh standalone -n '<Nifi Instance 1 IP>' -C 'CN=Admin1,OU=DevOU' -O -o /data/security


/data/download/nifi-toolkit-1.4.0/bin/tls-toolkit.sh standalone -n '<Nifi Instance 2 IP>' -C 'CN=Admin2,OU=DevOU' -O -o /data/security

This produced the standard files on each instance, which I have incorporated to the conf folder. The .p12 files I imported to my browser, and I can securely access the UI of the instances as Admin1 or Admin2 after modifying the authorizers.xml and authorzed-users.xml. (port 8443).

The S2S comms however is still giving me a headache.

The S2S properties are as follows. The difference between the two instances is the IP number. I understand that you shouldn't need to have the IP number on the RPG instance side, but it seemed to work with the numbers on both sides when I had Nifi unsecured, so i have left them in.

nifi.remote.input.host=<Nifi instance IP>
nifi.remote.input.secure=true
nifi.remote.input.socket.port=8899
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec

# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=8443
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200

I am using the 8899 port to have secure RAW protocol.

On instance 1 the RPG I have set the URL to be https://<instance 2 IP>:8899/nifi/ with RAW protocol.

I then imported the .p12 generated on each instance to the truststore of the other, using the keytool command

keytool -v
-importkeystore -srckeystore CN=Admin1_OU=DevOU.p12 -srcstoretype PKCS12
-destkeystore truststore.jks -deststoretype JKS

I now have an error on the RPG as follows:

I'm pretty sure the issue is authentication. My thinking is that the truststore of instance 1 needs the private key of instance 2 imported and vice versa. I had thought if I imported the .p12 files this would do the trick but it has failed.

I thought about the tutorial by Bryan and added users on both instances "CN=Admin1, OU=DevOU", "CN=Admin2, OU=DevOU" and even the one in the tutorial "CN=localhost, OU=NIFI" and gave each user policies for the site to site retrieval and added the users to the port policies too, but the error remains.

I've tried different combination of RAW and HTTPS in the S2S properties but they don't seem to help.

If I can get any further guidance, that would be appreciated.

Thank you

[#8]

When you configure the RPG, use the URL for NiFi that contains the port the API/UI is running on (in your case, 8443, not 8899). NiFi will negotiate the RAW/HTTP(S) S2S port automatically over that communication.

Do NOT import the private key of any server to any other server or truststore. The private key remains only on the instance you intend to secure with it. The public key can (and must) be imported to the other/remote truststores unless the key pairs were all signed by the same root, in which case the root public key alone in the truststore is sufficient.

[#9]

Hi Andy, thanks for your persistence.

Made the change to the URL to look at port 8443. Still same warning about the trust anchors.

I don't think I am importing the private key to the truststore of either instances. The .p12 file that the TLS Toolkit produces should be a certificate with the public key. It's the same file I import to a browser for it to access the Nifi UI. I don't understand why the nifi instances themselves have trouble authenticating.

is the root public key the .pem file that is created by the toolkit? the .key file appears to be the private key.

Thank you

[#10]

The .p12 is the client certificate and does contain the private key to identify the user (it must contain the private key in order to actually identify the user and perform any encryption/signing during TLS mutual authentication). Don't import the .p12 into any server; it is just for the browser to authenticate the user.

The public key of the CA is written out in nifi-cert.pem, and the various truststores from the host directories (i.e. <NiFi Instance 2 IP>/truststore.jks) contain that as a trustedCertEntry already. You can verify this by running keytool -list -verbose -keystore truststore.jks.

[#11]

Hi Andy,

Just letting you know that I got this working. Thank you for your help.

What I did was to exchange the two nifi-cert.pem files between the two nifi instances and import them in the opposing truststores.

At that point the RPG's status became "Forbidden".

I then looked at the nifi-users.log file on the target Nifi, and I could see that the there was a user with DN (CN=<Nifi Instance 1 IP>, OU=NIFI) that was trying to authenticate. So I added that user in the target nifi, added to the global S2S policy and the individual port policies and voila!

cheers