Member since
01-11-2017
17
Posts
2
Kudos Received
0
Solutions
05-16-2017
01:34 PM
@bhumi limbu That is a very useful ERROR log message. You appear to be running in to teh following reported issue: https://issues.apache.org/jira/browse/NIFI-3096 Thanks, Matt
... View more
03-01-2017
10:19 PM
@Matt Thank you it was very helpful.
... View more
01-31-2017
07:44 PM
1 Kudo
@bhumi limbu NiFi FlowFile attributes/metadata lives in heap. The List based processors return a complete listing from the target and then creates a FlowFile for each File in that returned listing. The FlowFiles being created are not committed to the list processor's success relationship until all have been created. So you end up running out of NiFi JVM heap memory before that can happen because of the size of your listing. As NiFi stands now, the only option is to use multiple list processors with each producing a listing of on a subset of the total files from your source system. You could use the "Remote Path", "Path Filter Regex" and/or "File Filter Regex" properties in the listSFTP to filter what data is selected to help reduce the heap usage.
You could also increase the available heap to your NiFi's JVM in the bootstrap.conf file; however, I find it unlikely considering the number of FlowFiles you are listing that you will not still run out of heap memory. I logged a Jira in Apache NiFi with a suggested change to how these processors produce FlowFiles from the listing returned by these types of processors: https://issues.apache.org/jira/browse/NIFI-3423 Thanks, Matt
... View more