SiteToSiteBulletin reporting Task 403:Forbidden with explanation: null Error

[#1]

Hi,

I have SiteToSiteBulletinReportingTask setup to capture bulletin information for a standalone secure nifi instance. the reporting task is pointing to an input port as expected but when I start the task it doesnt work and it keeps giving this warning:

SiteToSiteBulletinReportingTask[id=e99b03f1-0173-1000-c7ef-4e475ed2416d] org.apache.nifi.remote.client.PeerSelector@24677b11 Unable to refresh Remote Group's peers due to response code 403:Forbidden with explanation: null"

The reporting task is setup as follows:

The SSL context service is setup with the same values for the keystore and truststore as in the nifi.properties file which is working to secure my instance.

I have added the policy "receive data via site-to-site" to the input port "S2S_Bulletins_rec" .

The user created by securing the instance has the policy "retrieve site-to-site details".

The error doesn't give much details. Can you please help? Thank you.

[#2]

@SAMSAL 

NiFi Site-To-SIte (S2S) components perform to things:

1. A background process runs every 30 seconds which connects to the target URL entered (In this case https://localhost:9443/nifi ) to retrieve S2S details.  These details include details like how many nodes in target NiFi cluster, hostnames for those target NiFi nodes, load on those nodes, if those nodes support http and/or RAW transport protocols, What remote input ports exist that this source node is authorized to see, etc...  S2S details are always fetched over HTTP even if you set transport protocol to RAW.
2. Then the source NiFi uses this data to actually sent content over S2S to all the target nifi nodes in a distributed fashion.

Since your target is a https, the first thing that needs to happen is a Mutual TLS handshake.  That means the keystore configured in the SSL Context service must contain a PrivateKeyEntry that support an EKU with "clientAuth".  The target NiFi which is probably returned a FQDN via those S2S details, will also send its server certificate to the client.  That means the truststore configured in your sslContextService must contain the complete trust chain for that certificate.   That server certificate which comes from the keystore in the nifi.properties file on the target NiFi must contain a single PrivateKeyEntry with a EKU that supports "serverAuth" and must also have a SAN that matches the hostname used to connect with.  The target NiFi's truststore configured in its nifi.properties file must also contain the complete trust chain for that client certificate presented from the sslContextService.

So if all the above is properly in place, and I would guess it is since you are trying to S2S back to same NiFi cluster/instance and are probably using same keystore and truststore in your SSLContextService as is configured in the nifi.properties file, that those files are good.  I would however be concerned with your use of "localhost" as the target URL because I doubt the server certificate sent in that TLS handshake is going to have a SAN entry that contains "localhost".  You should instead provide the actual hostname for that target NiFi.  It is fine and common to use localhost in the "instance URL"field as that is only used to identify the host that sent the FlowFile to the target InputPort.

The only other statement that stands out to me is "The user created by securing the instance has the policy "retrieve site-to-site details"."  NiFi authentication and authorization is setup to control what users are allowed to do once they access a secured NiFi UI.  The components that are added by the authorized user do not execute as that authenticated user.  All components execute as the NiFi service user.  In that case of this S2S reporting task, it is executing as the NiFi service user but authenticating to the target through that mutual LS handshake, which means the DN form that clientAuth certificate is going to be the user that needs to be authorized for both the "retrieve Site-To-Site details" and "receive data via site-to-site" NiFi authorization policies.

I know there is a lot of information here and hope it is clear.

If you found this addressed your query, please take a moment to login and click accept on this solution.
Thank you,

Matt

[#3]

Hi Matt,

Thank you very much for taking the time and explaining how this process works. I spent a lot of time trying to troubleshoot the issue and I think I was able to resolve. what caught my eye in other similar post is the case of DN being authenticated and authorized which what you also highlighted "...which means the DN form that clientAuth certificate is going to be the user that needs to be authorized for both the "retrieve Site-To-Site details" and "receive data via site-to-site" NiFi authorization policies". Im not a security expert here but I assume DN refers to Domain, right? In any case for me to resolve this issue I had to do the following:

1- In the "SiteToSiteBulletinReportingTask" properties I kept the default value for "Instance URL" which is "http://${hostname(true)}:8080/nifi" despite my nifi instance is being secured. Im stil not very clear what this value is for and the tutorial doesnt give much info on and I always thought it has to match what is set for the Destination URL but its not.

2- In Nifi I have added the user "CN=localhost , OU=NIFI" and made sure to give it all kind of permission including view & modify data. Im thinking since my domain is "locahost" and the domain "DN" is what is being authenticated\authorized I needed to create this user account.

3- In the nifi.properties file I made sure the following properties are set as follows:

         nifi.remote.input.host=
         nifi.remote.input.secure=
         nifi.remote.input.socket.port=8080
         nifi.remote.input.http.enabled=true

    If I set the "nifi.remote.input.secure" to true then I will keep getting 403 Forbidden error  despite my instance is secured!

Not sure if all of the above helped or combination of it but at least its working and I'm able to replicate playing with those parameters. Im hoping someone from the Nifi community can maybe elaborate more on this to confirm as Im not a security expert. Thank you.

[#4 · Accepted solution]

@SAMSAL 

Sounds like you are very close...

The 403 has nothing to do with authentication, but rather authorization.
This eludes to the fact that the client was able to be properly trusted through the mutual TLS handshake, but when that resulting client string was checked against the endpoint policy being checked it was not present, thus resulting in a 403 response.

The DN is the DistinquishedName for your certificate.  You can use the below command to get the verbose details on the certs added to a keystore:

keytool -v -list -keystore <keystore.jks or truststore.jks>

 You will see that each certificate has either type "PrivateKeyEntry" or "TrustedCertEntry"
For each certificate you will see an "Owner" and an "Issuer"
For each of those it will show the complete DN which would look something like:
CN=<some string>, OU=<some string>, DC=<some string>

There are numerous parts that can be part (CN, OU, O, ST, DC, etc.) of a DN and various lengths 

The "Owner" DN for the PrivateKeyEntry is what is used as the client/user authenticated string after the successful mutual TLS handshake.  The nifi.properties file has optional ideNtity.mapping properties that can be configured to trim and manipulated these identity strings (for example pulling our on the string from the CN).

With nifi.remote.input.secure= set to false, FlowFiles send over S2S will not be over a TLS encrypted connection.  The fact that you say it works when you have this set to false and you can still successfully obtain S2S details from the configured https:// secured NiFi Destination URL tells me that authentication is correct for "Retrieve site-to-site details" NiFi policy.  That only leaves having incorrect policy setup for your remote input port.

You should open a command prompt on each node and "tail -F /<path to>/nifi-user.log". Then enable your controller services and check the logs being tailed for the authorization exception.  It should clearly show you the "string" being checked against the "receive data via site-to-site" endpoint which in the log would look like "/data-transfer/input-ports/<uuid of remote input port>".  Also remember that you are NOT authorizing your user to any of these S2S policies, but rather the DN or identity mapped DN string.

Really hope this helps you get fully secured here.
Matt