Dear @Tine,   I have exactly the same use case. Have you implemented the proposed solutions from above? I would really appreciate it if you could share your experience!   Thanks in advance! Valentina     ... View more

Thanks for the revert mate. Much Appreciated.  ... View more

@vivek12    Your last statement is confusing to me. Are you saying only the invokeHTTP processor is stopped after a NiFi restart?   This implies that someone stopped it or the flow.xml.gz is not getting updated with that processors last known state.  I'd inspect what is written in your flow.xml.gz on each node to make sure they all show the state as running. The last known state of a processor is not something that is checked as node join a cluster.  Nodes joining are checked to make sure their flow.xml matches the elected cluster flow and if it does, the node can join.  That node will then be told to start those components which are running in the cluster. When you restart the entire cluster each node presents its flow.  First node has Flow x and gets 1 vote, next node flow is checked and it matches exactly, that flow gets 2 votes.  Next node comes in and his flow is not exactly the same, so it gets 1 vote.  The flow with the most votes becomes the cluster flow.  If you have an even number of nodes (for example 4 nodes) and 2 node's flows get a vote and other 2 node's get a different vote.  since you have 2 vs 2, NiFi will end up picking one at random.  My concern here is that some node(s) have last known state as stopped while another is running.  So sometimes with a complete restart of your cluster you end up starting the flow with the stopped state on this processor. The other possibility is this invokeHTTP processor is failing validation on some node on startup and resulting in processor being stopped. Have you tried copying the flow.xml.gz from one node to all the other nodes?   Hope this helps, Matt ... View more

Can anyone share screenshot of the processor configuration to understand this solution. ... View more

@Rohitravi  The None of NiFi's processors will release any FlowFiles to a downstream connection until the end of the thread operation.  This is to protect users from dataloss and in some cases data duplication in the result of a failure. In the case of a SplitText processor you have configured to split on every 10 lines.  The processor will stream the content of the first 10 lines in to a content claim in the content_repository and create a new FlowFile record pointing at that claim.  The next 10 lines may or may not go into that same content claim and another FlowFile record is created. above process continues until all splits have been created.  Then The processor releases all FlowFile created to the downstream connection at the same time. NiFi does not guarantee FlowFile processing order. You can adding the FirstInFirstOutPrioritizer to the downstream connections to help with ordering some. Hope this helps, Matt ... View more

Is it possible to do something similar but receiving the Server, Port and Path where the files are? Because GetSFTP and ListSFTP can't receive any input ... View more

@krishnaraj_v13    The error output is telling you that your NiFi node(s) have not been granted the proxy policy in your NiFi Registry.  Your NiFi-Registry policies are managed locally within NiFi-Registry. Your NiFi is setup to use Ranger ti handle authorizations and i see you mentioned you granted your NiFi nodes /proxy in Ranger, but these policies only apply for NiFi and not NiFi-Registry. Based on the authorizers.xml shared from NiFi-Registry, I can see you defined your NiFi nodes as local users in the file-user-group-provider, but did not also configure those nodes in the file-access-provider.  The file-access-provider actually created the initial policies in the authorizations.xml file and assigns users to those created policies. Note: Both NiFi and NiFi-Registry will only create the users.xml and authorizations.xml files if they do not already exist.  So modifications to these providers in the authorizers.xml file will not result in modifications to these files if they already exist. To resolve the error you are seeing you need to login to your NiFi-Registry with your initial admin user and grant your NiFi nodes the the following policies: 1. "Can proxy user requests".  (solves current error) 2. "Can Manage buckets" --> Read.  (allows NiFi nodes to read buckets to see if new flow versions have been committed) Hope this helps, Matt ... View more

@Aminsh  I am not sure where your response fits in to this thread. Are you asking a new question here? I recommend you start a new thread if that is the case.   Thanks, Matt ... View more