@David Yu - If you are not writing any data in to the new directory, why the need to create it. NiFi FlowFiles consist of FlowFile attributes and FlowFile content. Each FlowFile will have a filename associated with it. Processors like PutHDFS are designed to operate against incoming FlowFiles and are designed to put a file to HDFS. Without a FlowFile the processor would not be triggered to connect to HDFS at all. It cab be configured to dynamically create missing directories when writing the first file to a new directory path. - If you need to create some directory structure on a regular interval (for example new directory each hour based on timestamp), You would need to use a GenerateFlowFile processor (running using cron scheduling strategy and creating a 0 byte File) that feeds a PutHDFS processor. The "Success" relationship from putHDFS should then feed to DeleteHDFS to remove the 0 byte FlowFile that was added to the newly created directory in HDFS. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Angana Saha - I looks like you are trying to use Java 11 to run your NiFi instance. NiFi 1.8 supports Java 8. As of Java 9 the java.se aggregate module does not include the Java EE APIs. This is why you are seeing the following: org.apache.nifi.StdOut java.lang.module.FindException:Module java.xml.bind not found - I suggest installing the latest version of Java 8 and see if NiFI launches fine for you then. There is on going work to add support for Java 9 - 11 in future releases of NiFi. - Thank you, Matt ... View more

The NiFi Expression Language function ${hostname()} does not return what is configured in the nifi.properties file for nifi.web.http[s].host. It uses java to query system for hostname. So an improperly configured system may return localhost as the hostname. But you are correct that it only returns the hostname of the NiFi node on which it was executed. I believe that is why he was querying the rest-api/controller/cluster endpoint to get json which includes all nodes in cluster and their current connection status. ... View more

@Adam J *** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion. - Even if you use PostHTTP to ListenHTTP and a particular FlowFile is destined for a specific node and that node goes down, that data will remain sitting in that connection queue unless you route "failure" relationship to another postHTTP with a different destination. And if postHTTP is running primary node only, again you will have data stuck on the old primary node if a primary node changes. Just because a primary node change occurs does not mean the node is necessarily down. - Resulting behavior seems same to me here. If a node goes down, all data on that node is stuck on that node until it is recovered. This includes FlowFiles on primary node feeding a postHTTP (if primary node goes down or changes) and FlowFiles redistributed via a connection using "Partition by Attribute" if one of the nodes receiving a partition of data goes down. - Unless you are regularly polling the nifi-api/controller/cluster endpoint to see what nodes are actually connected. and adjusting which nodes get data based on who is still connected. But then you end up with a node not getting some data or some other node maybe getting double data? - Would be interested in what your proposed dataflow design would look like. That may clear up some of my questions. - Thanks, Matt ... View more

@Adam J Please elaborate on your use case here. What do you mean by sending "request"? - Are you trying to have use postHTTP to send the same NiFi FlowFile to every node in your cluster? If so, then using PostHTTP to send to ListenHTTP is the correct path. I would not have PostHTTP configured to run on Primary node only. Only dataflow source processors should ever be configured fro primary node only operation. Having processors in the body of a dataflow configured for primary node only can result in data stuck in connection queues if a the elected primary node should change. - Thanks, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Suchet Renge There is no reason that you should not be able to auto-terminate a relationship via the rest-api. Are you seeing some error when trying to execute that specific rest-api endpoint? - Keep in mind that you can not Auto-terminate a relationship that is currently connected to another component. You must first either remove that relationship from any connection to which it has been assigned or delete the entire connection containing that relationship. - Here is a sample rest-api call to set relationship "success" to auto terminate on a logAttribute processor: curl 'http://<nifi-hostname>:<nifi-port>/nifi-api/processors/<processor UUID>' -X PUT -H 'Content-Type: application/json' --data-binary '{"component":{"id":"<processor UUID>","name":"LogAttribute","config":{"concurrentlySchedulableTaskCount":"1","schedulingPeriod":"0 sec","executionNode":"ALL","penaltyDuration":"30 sec","yieldDuration":"1 sec","bulletinLevel":"WARN","schedulingStrategy":"TIMER_DRIVEN","comments":"","runDurationMillis":0,"autoTerminatedRelationships":["success"]},"state":"STOPPED"},"revision":{"clientId":"9ef6f9e3-0168-1000-82cb-02214bcbc612","version":10}}' --compressed - Thank you, Matt ... View more

@Adam J By secure, I mean when NiFi has been configured to run over https. - In the invokeHTTP processor, why not just use NiFi Expression Language (EL) to build your remoteURL. for example: http://${hostname(true)}:9090/. This would allow which ever node is primary node to fetch its hostname and use it in the URL when the invokeHTTP executes. ... View more

@Adam J - Now i am following. You have a GetHTTP or invokeHTTP processor running on this NiFi cluster in primary node only mode which is configured with a url that is for one of the nodes within this same cluster. So this is not directly related to primary node at all. Your issue is what to do if the target node URL being used happens to become unavailable (down node). In that case you certainly could use localhost:8080. - Now if you decide at some point to secure your NiFi, localhost is not going to work. Another option may be to consider placing a Load-balancer in front of your NiFi to handle a down node scenario. That way target URL could be that of the LB which would be able to handle sending request to any one of the still up and running servers. - Thanks, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more