I have set up a simple flow in NiFi 1.1.2: ListFile -> (success) -> FetchFile ListFile reads /vagrant/testData folder. Running on primary NiFi node only. I have not changed any different settings than these two (directory to observe, execution scheduling) FetchFile moves the file to /vagrant/testData2. That's all. To reproduce the bug(?): Start the flow, do not have any files in the input directory Put a file into input directory File will be moved as expected -- this is correct behavior so far Move another file into the directory (different file name) You can wait 5 minutes, but the file will not be found by the ListFile processor anymore Stop ListFile processor Change a setting, for example "Recurse Subdirectories" to false Start ListFile processor again The second file you put into the input directory will now be moved to output directory Put a third file into the input folder --> nothing will happen again. Is this a known issue? How to fix it? ... View more

I have a NiFi cluster and want to process files that I store in a directory on one node to be distributed in the cluster. To keep the question short: Input files: 20ish files are coming in a batch each time The data flow works fine when I run it on one node (not distributed) The output will be stored on HDFS, so that's not a problem. Just that my input files will be distributed evenly to the NiFi nodes so the processing will be fast. How can I have it that the number of input files get split somewhat evenly on the NiFi nodes and each runs the entire flow for each file one by one? What I mean is that one of my processors need the entire file to be present on the node not just parts of the file. Should I upload all my input files to the primary node? How do files get distributed? A downside of this would be that all files need to be distributed in between nodes (this is doable though, the files are not that big). Another idea would be to store the input files in a specific directory on each node in the cluster. But then the "import" script that puts the files would need to know all nodes in the cluster and know about downtimes of the nodes as well... Thanks in advance ... View more

We are running NiFi instances "next to" Hortonworks HDP. This means we manually installed NiFi on our cluster nodes and let them connect to our ZooKeeper. What are the benefits of installing HDF next to HDP on our nodes when we only use NiFi and no other components such as MiNiFi or Kafka at the moment we use NiFi to process incoming files (files the get stored in a specific folder), transform these files, put them into HDFS, create logs about the transformation process, finish data flow after the transformed files are stored in HDFS they get analyzed in some way Should we consider using HDF instead of "plain" NiFi? Is it okay to run NiFi instances next to the HDP, without using HDF? Which problems can we run into? Thanks! ... View more