Hi All, One of our NiFi dataflows ingests small files (1-5 KB each) at a rate of 100+ messages per second. The requirement is to store them in HDFS, we're using a MergeContent processor to bundle 1000 files into a new bigger file, which makes the files a bit bigger for HDFS, but not near the ideal size for HDFS storage. We could make the MergeContent wait for more files until the desired-sized merged file is ready, but the problem is, we do not want to wait too long within NiFi, we want to be able to send data to HDFS as close to "near real-time" as possible, and not wait a day in MergeContent processor, waiting for enough files to accumulate. So, it appears PutHDFS "append" might work, where you write the files as they come in and append them to existing HDFS file until the desired HDFS file size is accomplished (have some questions on this approach, posted a question on that here - https://community.hortonworks.com/questions/99843/questions-on-nifi-puthdfs-append-option.html) Another option, we're considering is, have a nightly job/dataflow that merges HDFS files at rest to desired size; this seems like a simpler approach. Wanted to know which option would be better to address the too-many-small-files issue. Thanks in advance. ... View more

Hello, I have couple of questions on how NiFi's PutHDFS "append" option works. But let me start with a background before I ask the questions: Below is a schematic of how I'm envisioning PutHDFS append might be used, in a scenario where you have small files that are ingested and the requirement is to write them to HDFS as soon as the data comes in (for "near" real-time analysis of that data); and as we know that creates too many small files in HDFS, and the overhead and performance issues that result from that; So, one option, I think, is to write the small files to HDFS as they arrive, but keep appending to an existing HDFS file until you've grown that file to a decent size for HDFS; then start appending to a new HDFS file. With this approach, my understanding is that we need to keep checking the HDFS file size, to decide if it's now time to start writing/appending to a new HDFS file. Please see the schematic below. 1) In this approach, does the constant checking of HDFS file size create enough of an overhead to affect the performance of the dataflow or it's not that big a factor ? 2) Once we reached the desired size on the HDFS file, how does the file "close" happen ? does the PutHDFS processor (that has "append" selected for "Conflict Resolution Strategy"), after some idle time go back and close the file ? or does PutHDFS close the file after each append; if the latter (opening and closing for each append), then does that create overhead and degrade performance ? If I have it all wrong about how to use the PutHDFS append option, please let me know that as well. Thanks for your time. ... View more

Hello, I have couple of questions on data recovery/integrity after a standlaone NiFi instance crashes (and is subsequently restarted); I searched here in HCC about NiFi fault-tolerance, the few posts that I looked at talk about fault-tolerance and recovery in a cluster environment, but I'm interested in a standalone NiFi scenario. In our case the standalone NiFi instance crashed because the Java heap ran out of space (but we do have enough heap space allocated); at the time of the crash one dataflow was running a real-time stream without any issues, but when a second dataflow, that had ListSFTP and FetchSFTP processors, started running (there is a known issue with running List and Fetch processors when there are lots of files to process - https://issues.apache.org/jira/browse/NIFI-3423), some of the NiFi processors in the first dataflow started to throw out-of-memory errors and the ListenTCP processor stopped ingesting new flowfiles from the source system; and, on the server the Java process CPU utilization was something like 250%; at that point, we stopped both dataflows (the NiFi canvas was still accessible, it did not crash) and restarted the NiFi instance; after that, we resumed the first dataflow and all was fine; Since NiFi writes flowfile content to content repository and keeps attributes and state info in the flowfile repository, no data should have been lost or corrupted when NiFi instance crashed. Just wanted to clarify if my understanding in this scenario (no loss of data or integrity issues) is correct. 2nd question is, during a planned server reboot, if we stop a dataflow when there is data in transit in the dataflow (i.e. some flowfiles are in queues between processors, some flowfiles are being processed by NiFi processors, e.g. replacetext), then restart the NiFi instance and resume the stopped dataflow, would NiFi pickup where it was left off? i.e, flowfiles that were in the middle of being processed (just prior to the dataflow being stopped) resume their processing from where they were left off? my understanding is yes, that's how NiFi works, the fault tolerance built into NiFi takes care of that. Would like to know if that is correct ? Thanks in advance. ... View more