Problem in Establishing Site2Site Reporting task

[#1]

Dear NiFi Team,

                             I’m trying to Send Bulletin board info to mail using site to site reporting task.

Where I’ve used input port and sending to PutEmail Processor for the FlowFile , upon creating  the Site to site Reporting task and our NiFi is on Kubernetes Cluster.

While establishing this getting ended up with Warning message.  Suspecting are there any policies need to be added to this

Could you please help in establishing this task. I’ve attached all the evidences/configurations with screenshots.

By default its connecting to primary node. As mentioned in the ReportingConfigure mentioned both Destination URL and instance URL as same  value by following the https:// [Full DNS of cluster]:PortNumber. For your info. Cluster is of 3 nodes and our NiFi is of version 1.15.3

   

  

Please do let me know if you need any further info.

Thanks!!

[#2 · Accepted solution]

@PradNiFi1236 
 
There are numerous steps in this process.  So lets start with some basics on Site-To-Site (S2S).  The S2SBulletinReportingTask works much like the NiFi Remote Process Group (RPG).  When configured with a Destination URL and enabled a background thread will run independently on an interval to fetch S2S details from the destination URL.  If the destination url is a node in a NiFi cluster, the returned S2S details will include the hostnames of all the nodes in the cluster, whether cluster nodes are configured to support RAW and/or HTTP transport protocols, the configured RAW port for each node, node load average, etc...

Configuring just one destination URL from the target cluster does not change this behavior.  Configuring a comma separate list of nodes from the same destination cluster affords you HA.  If S2S details can't be retrieved from node URL 1, it then tries second URL, and so forth.

Also keep in mind it does not matter what node URL of a NiFi cluster you are accessing, any component (processor, reporting task, controller service, etc) added to the canvas is replicated to all nodes in the NiFi cluster.   So when you enable this S2SBulletinReporting task, all nodes are going to try to fetch S2S details.  Each node in a NIFi cluster has all the same components and executes all the same components (with exception of processors that can be scheduled to execute on primary node only). This means that all node will be trying to send generated bulletins to your cluster nodes.

So by what you shared, it looks like the background thread that fetches those S2S details is failing due to timeout.  This could be for any number of reasons.  Your configured keystore in the sslContextService does not contain a single PrivateKeyEntry that can be trusted by the truststore configured in the nifi.properties file on all 3 of your destination nodes.  The PrivateKeyEntry presented by the 3 NiFi nodes to the controller service is not trusted by what exists in the truststore configured in your SSLConTextService.  keystore used in sslContextService does not have a clientAuth PrivtaeKeyEntry in it. nifi.remote.input.secure is not set to true. nifi.remote.input.http.enabled not set to true.  

There are several authorization policies in play here as well, but I don't think you have even gotten that far yet.
Retrieve site-to-site details  <-- The privateKeyEntry from the keystore configured in the sslContextService will need to be authorized to retrieve these S2S details.  The keystore used in each of your 3 NiFi node's sslContext service may have unique DNs for their PrivateKeyEntry.  So all three of those unique keys would need to be authorized
Receive data via site-to-site  <-- The privateKeyEntry from the keystore configured in the sslContextService will need to be authorized to retrieve these S2S details.  The keystore used in each of your 3 NiFi node's sslContext service may have unique DNs for their PrivateKeyEntry.  So all three of those unique keys would need to be authorized.  This allows the S2SBulletinReporting task to see this bulletinmonitoring remote input port as an option to send bulletins to.

But if you were getting past authentication and failing on authorization, your exception would be different.  Instead of timeouts, you would be seeing not authorized exceptions.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

[#3]

@MattWho , Could you please help on this

[#4 · Accepted solution]

@PradNiFi1236 
 
There are numerous steps in this process.  So lets start with some basics on Site-To-Site (S2S).  The S2SBulletinReportingTask works much like the NiFi Remote Process Group (RPG).  When configured with a Destination URL and enabled a background thread will run independently on an interval to fetch S2S details from the destination URL.  If the destination url is a node in a NiFi cluster, the returned S2S details will include the hostnames of all the nodes in the cluster, whether cluster nodes are configured to support RAW and/or HTTP transport protocols, the configured RAW port for each node, node load average, etc...

Configuring just one destination URL from the target cluster does not change this behavior.  Configuring a comma separate list of nodes from the same destination cluster affords you HA.  If S2S details can't be retrieved from node URL 1, it then tries second URL, and so forth.

Also keep in mind it does not matter what node URL of a NiFi cluster you are accessing, any component (processor, reporting task, controller service, etc) added to the canvas is replicated to all nodes in the NiFi cluster.   So when you enable this S2SBulletinReporting task, all nodes are going to try to fetch S2S details.  Each node in a NIFi cluster has all the same components and executes all the same components (with exception of processors that can be scheduled to execute on primary node only). This means that all node will be trying to send generated bulletins to your cluster nodes.

So by what you shared, it looks like the background thread that fetches those S2S details is failing due to timeout.  This could be for any number of reasons.  Your configured keystore in the sslContextService does not contain a single PrivateKeyEntry that can be trusted by the truststore configured in the nifi.properties file on all 3 of your destination nodes.  The PrivateKeyEntry presented by the 3 NiFi nodes to the controller service is not trusted by what exists in the truststore configured in your SSLConTextService.  keystore used in sslContextService does not have a clientAuth PrivtaeKeyEntry in it. nifi.remote.input.secure is not set to true. nifi.remote.input.http.enabled not set to true.  

There are several authorization policies in play here as well, but I don't think you have even gotten that far yet.
Retrieve site-to-site details  <-- The privateKeyEntry from the keystore configured in the sslContextService will need to be authorized to retrieve these S2S details.  The keystore used in each of your 3 NiFi node's sslContext service may have unique DNs for their PrivateKeyEntry.  So all three of those unique keys would need to be authorized
Receive data via site-to-site  <-- The privateKeyEntry from the keystore configured in the sslContextService will need to be authorized to retrieve these S2S details.  The keystore used in each of your 3 NiFi node's sslContext service may have unique DNs for their PrivateKeyEntry.  So all three of those unique keys would need to be authorized.  This allows the S2SBulletinReporting task to see this bulletinmonitoring remote input port as an option to send bulletins to.

But if you were getting past authentication and failing on authorization, your exception would be different.  Instead of timeouts, you would be seeing not authorized exceptions.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

[#5]

Thank you so much @MattWho  For taking time and looking into the issue. Whatever the improvements that you suggested which were already there for site to site details in all 3 nodes.

# Site to Site properties
nifi.remote.input.host=#################
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10000
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30

secnifi.remote.contents.cache.expiration=30 secs

and both ServerAuth and ClientAuth privatekey Entitiy allowed in all 3 nodes. 

as you are telling Authorization Exceptions. I've experienced those errors too earlier before when tried to call one node from other node (Unable to refresh remote group peers due to: response code 401:Unauthorized with explanation: null), 

Please let me know if this error is specific to certificate which i'm referring to using SSL/is this due to the Destination URL which i'm using / are there any specific properties which i need to look in?

Thank you,

Pradeep

[#6]

@PradNiFi1236 

What do you see in the nifi-user.log on the NiFi receiving the FlowFiles from this reporting task?
I would expect it to show the 401 and hopefully the identity string that was used to identify the client.

Matt

[#7 · Accepted solution]

@PradNiFi1236 
The Remote input port will use the keystore and truststore configured in the nifi.properties file.
The S2SBulletinReportingTask will use the keystore and truststore configured in the SSLContextService Controller service.

It would be difficult for me to help with a potential SSL Handshake issue without the verbose output of those 4 files that are being used.

<path to>/keytool -v -list -keystore <keystore or truststore filename>

You need to verify that the compleete trust chain exist in the truststore used in the nifi.properties file for the ClientAuth PrivateKeyEntry from the keytsore configured in the SSLConextService.

You need to verify that the complete trust chain exist in the truststore used in the SSLContextService for the ServerAuth PrivateKeyEntry found in the keystore from the nifi.properties file.

You also need to make sure that your keystore does not contain more than 1 PrivateKeyEntry in it.

You need to make sure that the PriavteKeyEntry has correct SAN entry(s).

You should tail the nifi-user.log on the host configured in the S2SBulletinReportingTask and then enable that reporting task.  If the MutualTLS handshake was successful, you should see the request being made for the S2S details.  This would help you understand the exact client identity string that is being checked for authorization to /site-to-site (pretty name for policy: Retrieve site-to-site details) NiFi resource identifier policy.

I also don't know the full destination URL you have configured to verify it is correct. It should just be: 

https://<nifihostname>:<nifiport>/

Where <nifiport> is the same port you use to access the web UI canvas.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

[#8]

Thanks for the Suggestions @MattWho . Was able to sort of the authentication issue and i also  tailed the nifi-user.log , in that observed authentication successful. But now I'm observing a different error. the input port name  which I've given in Configure Reporting Task settings even though my flow is root of the canvas.

Could you please help on this issue.

[#9]

Hi @MattWho , After i Enabled Retrieve site-to-site details for the Input port BulleInfo, then i was able to receive the BulletinInfo.

Thank you so much for you suggestions!!

Regards,

Pradeep

[#10]

@PradNiFi1236, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.