Understanding Content Repository Cleanup and Retention Behavior in Apache NiFi 1.0

Question

Hello,

First time posting here so sorry if this is in the wrong section / wrong format.

First, some background : We started a POC using NiFi 1.0.0. We're using a 3 node cluster with limited ressources (this is a POC...). Each of the node has 16 cores, 32gb of ram and 2 volumes : a raid 1 volume for the OS and a Raid 10 volume on 2.5in splindles. I know this is not a recommended setup but the content repo, the provenance repo, the flow files, everything basically, is on the same raid 10 array. The disks are heavily used right now. Content Repo archiving is disabled.

Now here's the thing : every 2-3 days, the disk fills up. Right now, the UI reports that we have, in queue : 450 000 (3.21gb). I would expect to have roughly the same amount of data in the nifi/content_repository folder but it's not the case : On one of the node, the content_repo folder is 73gb. I can't tell how big the 2 others nodes are since the "du -h" operation is still running after 10minutes but using "df", I can estimate around 700-800gb on each.

When we restart one of the node, it can take hours while the process cleans the content_repo and spams the log with a bunch of "unknown files"

Any ideas / Suggestions? This is running on CentOS 6.

Thanks

Here's the relevant config section :

nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.repository.directory.default=./content_repository nifi.content.repository.archive.max.retention.period=1 hours nifi.content.repository.archive.max.usage.percentage=1% nifi.content.repository.archive.enabled=false nifi.content.repository.always.sync=false

MattWho · Accepted Answer

@Philippe MarseilleThe content size displayed in a the UI will not map exactly to disk utilization since Nifi stores multiple FlowFiles in a single claim in the content repo.  A claim cannot be deleted until Every FlowFile in contains has reached a point of termination in your dataflow. so it is possible with 450,000 queued FlowFiles you are holding on to a large number of claims still.  Try clearing out some of this backlog and see if disk usage drops.  Setting backpressure thresholds on connections is good way to prevent your queues from getting so large.Another possibility is that you are running in to https://issues.apache.org/jira/browse/NIFI-2925 .This bug has been addressed for the next Apache NiFi release of 1.1 and HDF 2.1Thanks,Matt

AceWinner · Answer

Thanks @Matt, Clearing the queues does not seem to help. I'm restating one of the nodes right now, I'll be able to share the exact message when it boots and discovers the files that should not be there - sounds a lot like we're hitting the bug.

Is there a timeline for the release of 1.1.0? Reading the mailing lists, it seems to be really close to RC.

Thanks

Phil

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.