Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi dataflow is exceeding provenance recording rate

Nifi dataflow is exceeding provenance recording rate

New Contributor

I keep getting "nifi dataflow is exceeding provenance recording rate slowing down flow to accommodate". I currently have nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository set. Would setting it to org.apache.nifi.provenance.VolatileProvenanceRepository have any benefits and what are cons of setting it to volatile, loss of data?

2 REPLIES 2

Re: Nifi dataflow is exceeding provenance recording rate

@Derek Calderon

Change the following property to the value shown: nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository

This has been available since NiFi 1.2.

Note: If implemented, then comment out this line in the bootstrap.con file, java.arg.13=-XX:+UseG1GC,

there are some know issues which could cause corruption in JVM heap.

Re: Nifi dataflow is exceeding provenance recording rate

Master Guru

@Derek Calderon

Just to add to the best recommend option made by @Wynner above, the PersistentProvenance implementation is old and known to be slow when dealing with high volumes of events. The WriteAheadProvenance implementation if the much faster replacement. Are you positive current configuration is using WriteAhead? Was NiFi restarted after making the change?

If so, consider increasing these provenance configuration property:
nifi.provenance.repository.concurrent.merge.threads


You may also want to look into if disk I/O is an issue. Have you configured the NiFi provenance, content, and flowfile repositories to each have their own dedicated disks?



The VolatileProvenance implementation is held in the NiFi JVM's heap memory space. Setting the number of retained events too high can lead to OOM issues in NiFi. In addition, since it is held in heap, all events are list anytime the NiFi JVM restarts.

Don't have an account?
Coming from Hortonworks? Activate your account here