You should be able to have the NiFi on your local machine pull from the NiFi on the remote machine... Remote machine would have ListHDFS -> FetchHDFS -> Output Port Local machine would have Remote Process Group pointing to remote NiFI, and then the connection from the output port to whatever you want to do locally. The remote NiFi will need site-to-site enabled by setting nifi.remote.input.socket.port and that port will also need to be open through the firewall. ... View more

Can you give some background on why you want to set the classpath? NiFi has a very specific class-loader isolation model and generally you don't add JARs to NiFi's classpath like you do with other projects. ... View more

1) The Remote Process Group checks periodically (I think once per minute) with the NCM to get the status of the nodes in the cluster. If NCM is down and then one of the other nodes fails, the primary node will try to send data to that failed node, but it will get some kind of exception and then it will move on and try another node. So primary node doesn't know the other node is dead, but keeps trying to nodes until on succeeds. 2) Yes if you run the cache server on the NCM and NCM fails, then the other nodes can't access the cache server. The long term solution is to use a true distributed cache (memcached, redis, etc) as the backing implementation that the cache client talks to, this just hasn't been implemented yet. ... View more

You probably have a couple of options... I don't think you want the same NiFi instance that is running your main dataflow, also using PutSplunk to monitor itself. If you had TailFile -> PutSplunk, where TailFile was tailing the same instance, it would potentially create a cycle where the more your tailed and sent to splunk, the more logs produced, the more you tailed, the more logs you produced, etc. I would suggest a second NiFi instance (maybe even the MiNiFi Java agent) to monitor the logs of the main instance. Another possibly simpler solution... configure the NiFi logback.xml to add a UDP/TCP appender that can send logs to Splunk. This way anything NiFi logs to nifi-app.log will get forwarded to Splunk. Last option, slightly different than logging, NiFi has a concept called a ReportingTask that can be used send metrics and statistics to other systems. If that was the information you were interested in, you could implement a custom ReportingTask to send data to Splunk. ... View more