Hello, I've NiFi (standalone instance 1.0.1) GetHDFS with this cron schedule - 0 30 0 * * ? I want the processor to start at 12:30 AM daily; so, with the above schedule, the processor started at the expected time this morning and some files have been read, but it hasn't finished reading all the files; I had quite a few files to read in the directory yesterday and right now it still has 1200+ files left in the directory; I have the "Keep Source File" set to false, so it would/should delete the files as it reads; that shows the files left in the directory haven't been read by the processor; My understanding is, with the above schedule, once GetHDFS starts, it should keep reading until all the files in the directory are exhausted; but I'm not understanding why some files are still left. Please help, thank you. ... View more

Hi All, I'm running into an issue while trying to merge large number of small files in NiFi; I've about 800K files (350 MB) in the queue at MergeContent processor; I'm waiting to accumulate about 1.2 mil files, to merge them into 1 large file; but the MergeContent processor is throwing this error below; MergeContent[id=3104122b-1077-115c-2e71-b264709ceb44] Failed to process bundle of 897788 files due to org.apache.nifi.processor.exception.FlowFileAccessException: Failed to read content of StandardFlowFileRecord[uuid=a2a32c84-f633-4a7a-8b82-2ba5547db9af,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1498156308912-3769, container=default, section=697], offset=429054, length=436953],offset=104885,name=9b425a01-a759-42b6-bcf6-67f9bc79c871,size=302]; rolling back sessions: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to read content of StandardFlowFileRecord[uuid=a2a32c84-f633-4a7a-8b82-2ba5547db9af,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1498156308912-3769, container=default, section=697], offset=429054, length=436953],offset=104885,name=9b425a01-a759-42b6-bcf6-67f9bc79c871,size=302] 2017-06-22 13:37:49,515 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: java.io.FileNotFoundException: /data1/apache-nifi/content_repository/676/1498156300076-3748 (Too m any open files) 2017-06-22 13:37:49,516 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.io.FileInputStream.open0(Native Method) 2017-06-22 13:37:49,516 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.io.FileInputStream.open(FileInputStream.java:195) 2017-06-22 13:37:49,516 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.io.FileInputStream.<init>(FileInputStream.java:138) I'm thinking that it's suggesting that I'm over some kind of threshold; Would you please let me know which of the content repository properties I should increase, to allow more files to wait in the queue, to be merged. nifi.properties: # Content Repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.claim.max.appendable.size=10 MB nifi.content.claim.max.flow.files=100 # nifi.content.repository.directory.default=./content_repository nifi.content.repository.directory.default=/data1/apache-nifi/content_repository nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false nifi.content.viewer.url=/nifi-content-viewer/ ... View more

Hi All, I'm trying to unpack/de-merge merged text files that were merged earlier using the "Binary Concatenation" format; in the UnpackContent processor's "Packaging Format" property, there is no "Binary Concatenation" format available; by the way we're on NiFi 1.0.1; Is there a workaround ? Thanks. ... View more