Member since
04-29-2016
192
Posts
20
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1507 | 07-14-2017 05:01 PM | |
2550 | 06-28-2017 05:20 PM |
06-26-2017
01:41 PM
@Shashank Chandhok the schedule change to "0,30 30 0 * * ?" helped to read few additional files, but many files still remain in the directory
... View more
06-25-2017
03:39 PM
Hello, I've NiFi (standalone instance 1.0.1) GetHDFS with this cron schedule - 0 30 0 * * ? I want the processor to start at 12:30 AM daily; so, with the above schedule, the processor started at the expected time this morning and some files have been read, but it hasn't finished reading all the files; I had quite a few files to read in the directory yesterday and right now it still has 1200+ files left in the directory; I have the "Keep Source File" set to false, so it would/should delete the files as it reads; that shows the files left in the directory haven't been read by the processor; My understanding is, with the above schedule, once GetHDFS starts, it should keep reading until all the files in the directory are exhausted; but I'm not understanding why some files are still left. Please help, thank you.
... View more
Labels:
- Labels:
-
Apache NiFi
06-24-2017
10:46 PM
@Pierre Villard, chaining 2 MergeContent Processors, as you suggested, worked for me; thank you.
... View more
06-23-2017
01:07 PM
@Sonu Sahi thanks; I'm going to try what @Pierre Villard suggested first, before I go this route.
... View more
06-22-2017
08:16 PM
1 Kudo
Hi All, I'm running into an issue while trying to merge large number of small files in NiFi; I've about 800K files (350 MB) in the queue at MergeContent processor; I'm waiting to accumulate about 1.2 mil files, to merge them into 1 large file; but the MergeContent processor is throwing this error below; MergeContent[id=3104122b-1077-115c-2e71-b264709ceb44] Failed to process bundle of 897788 files due to org.apache.nifi.processor.exception.FlowFileAccessException: Failed to read content of StandardFlowFileRecord[uuid=a2a32c84-f633-4a7a-8b82-2ba5547db9af,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1498156308912-3769, container=default, section=697], offset=429054, length=436953],offset=104885,name=9b425a01-a759-42b6-bcf6-67f9bc79c871,size=302]; rolling back sessions: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to read content of StandardFlowFileRecord[uuid=a2a32c84-f633-4a7a-8b82-2ba5547db9af,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1498156308912-3769, container=default, section=697], offset=429054, length=436953],offset=104885,name=9b425a01-a759-42b6-bcf6-67f9bc79c871,size=302] 2017-06-22 13:37:49,515 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: java.io.FileNotFoundException: /data1/apache-nifi/content_repository/676/1498156300076-3748 (Too m
any open files)
2017-06-22 13:37:49,516 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.io.FileInputStream.open0(Native Method)
2017-06-22 13:37:49,516 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.io.FileInputStream.open(FileInputStream.java:195)
2017-06-22 13:37:49,516 ERROR [NiFi logging handler] org.apache.nifi.StdErr at java.io.FileInputStream.<init>(FileInputStream.java:138)
I'm thinking that it's suggesting that I'm over some kind of threshold; Would you please let me know which of the content repository properties I should increase, to allow more files to wait in the queue, to be merged. nifi.properties: # Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
# nifi.content.repository.directory.default=./content_repository
nifi.content.repository.directory.default=/data1/apache-nifi/content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/
... View more
Labels:
- Labels:
-
Apache NiFi
06-22-2017
04:20 PM
@Bryan Bende thank you, I'm trying with SplitContent, initial results are encouraging.
... View more
06-21-2017
06:51 PM
Hi All, I'm trying to unpack/de-merge merged text files that were merged earlier using the "Binary Concatenation" format; in the UnpackContent processor's "Packaging Format" property, there is no "Binary Concatenation" format available; by the way we're on NiFi 1.0.1; Is there a workaround ? Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
05-31-2017
07:58 PM
@Joshua Adeleke I had somewhat similar issue, the State of client service would get stuck on "Disabling" and the edit option would disappear; restarting the NiFi instance brought back the Edit option for me.
... View more