It's really just the same message as the original.  (Un)fortunately, 2 of the 3 nodes of the cluster just dropped a few min ago so: 7/11/2024 16:57:12 UTC: Node disconnected due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption. 07/11/2024 16:57:12 UTC: Disconnection requested due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption. 07/11/2024 16:57:07 UTC: Node Status changed from CONNECTED to DISCONNECTED due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption. 07/11/2024 16:56:56 UTC: Received first heartbeat from connecting node. Node connected. 07/11/2024 16:56:54 UTC: Requesting that node connect to cluster ... View more

@MattWho  Thank you, .. adding the username transformer did the trick. Had a guess that the transformer might be the case, but could not find upstream documentation on what to key to update when using USE_USERNAME ... View more

Yes, almost same behavior is observed with retry strategy as "penalize". Just the additional penalty duration gets added into the time. For example by default the penalty duration is 30 secs, if incoming flow files are 10 and number of retries is 1. For this case 10 flow files are clubbed up and first retry happens at 50secs. Then for 30secs it penalizes the clubbed flow files. Then after 50secs it goes into the failure relationship. So, In total (numberOfRetries+1)*5secs*(numberOfInComingFlowFiles) + Penalty duration time taken by publishKafka processor to route file into failure relationship in case of penalize retry policy.   If retry is not checked then similar behavior like yield is observed 5*numberOfIncomingFlowFiles secs to route to failure relationship as shown in photos. Penalty and yield settings are default only. target kafka version is  3.4.0 and number of partition is 1. Number of nifi nodes are 3. Number of concurrent Tasks on PublishKafkaRecord is 1, but the execution is on all nodes, which is I think 1 thread on 3 nodes each.   ... View more

@enam Have a slight mistake in my NiFi Expression Language (NEL) statement in my above post. Should be as follows instead: Property = filename Value = ${filename:substringBeforeLast('.')}-${UUID()}.${filename:substringAfterLast('.')} Thanks, Matt ... View more

@Vikas-Nifi  the following error is directly related to failure to establish certificate trust in the TLS exchange between NiFi's putSlack processor and your slack server: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target " The putSlack processor utilizes the StandardRestrictedSSLContextService to define keystore and truststore files the putSlack processor will use.  The truststore must contain the complete trustchain for the target slack server's serverAuth certificate. You can use: openssl s_client -connect <companyName.slack.com>:443 -showcerts to get an output of all public certs included with the serverAuth cert. I noticed with my slack endpoint that was not the complete trust chain (root CA certificate for ISRG Root X1 was missing from the chain). You can download the missing rootCA public cert directly from let's encrypt and add it to the truststore set in the StandardRestrictedSSLContextService. https://letsencrypt.org/certificates/ https://letsencrypt.org/certs/isrgrootx1.pem https://letsencrypt.org/certs/isrg-root-x2.pem You might also want to make sure all intermediate CAs are also added and not just the intermediate returned by the openssl command just in case server changes that you get directed to. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@greenflag  Not knowing anything about this rest-api endpoint, all I have are questions. How would you complete this task outside of NiFi? How would you accomplish this using curl from command line? What do the REST-API docs for your endpoint have in terms of how to get files?  Do they expect you to pass the filename in the rest-api request? What is the rest-api endpoint that would return the list of files? My initial thought here (with making numerous assumptions about your endpoint) is that you would need multiple InvokeHTTP processors possibly.   The first InvokeHTTP in the dataflow hits the rest-api endpoint that outputs the list of files in the endpoint directory which would end up in the content of the FlowFile.  Then you split that FlowFile by its content so you have multiple FlowFiles (1 per each listed file).  Then rename each FlowFile using the unique filename and finally pass each to another invokeHTTP processor that actually fetches that specific file. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@NeheikeQ  yes, newer version of 1.x NiFi-Registry will support older versions of NiFi version controlling to it. For NiFi after upgrade, load the flow.xml.gz on one node and start it.  Then start the other nodes so that they all inherit the flow from the one node where you had a flow.xml.gz.   At this point all nodes should join successfully and will have the same dataflow loaded.  Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@SAMSAL @cotopaul @MattWho : please help on above ... View more

That's a good idea, however low latency is a user requirement.  Currently, processing each file from source to destination takes around one minute.  If I add a two minute delay, the users would not be happy. ... View more