question Re: HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB in Support Questions

HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB

DivyaKaki — Thu, 22 Oct 2020 16:14:53 GMT

HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB

I have a requirement to retain provenance for 5 days and made necessary changes

- provenance retaining hardly - 2 days and less

content_repo - 500GB utilization 10%
Provenence_repo - 500GB Utilization 98%
flowfile_repo - 500GB utilization 10%

below are the configs

@MattWho @TimothySpann Please advice

Re: HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB

TimothySpann — Thu, 22 Oct 2020 16:23:30 GMT

you need to restart your servers after making the setting changes. do you have space?

Re: HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB

DivyaKaki — Thu, 22 Oct 2020 16:32:57 GMT

@TimothySpann thanks for the update. restarted the cluster 2 weeks back after making changes.
but still, the Provenance repo is piling up. surprised to see that huge disc being filled by provenance

Re: HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB

TimothySpann — Thu, 22 Oct 2020 16:47:31 GMT

provenance is for every step of every flowfile in every flow.

I recommend rolling those logs

Re: HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB

DivyaKaki — Thu, 22 Oct 2020 17:56:17 GMT

Hi Tim,

Are you recommending to role nifi provenance? could you provide more pointers?

Re: HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB

TimothySpann — Thu, 22 Oct 2020 18:12:09 GMT

https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html

Maybe shrink your provenance repo and send everything you need via:

https://www.datainmotion.dev/2020/04/sql-reporting-task-for-cloudera-flow.html

Or via Atlas

Re: HDF 3.4.1 NIFI 1.9 - NIFI Provenance Repository filling disc 500GB

DivyaKaki — Thu, 22 Oct 2020 18:18:32 GMT

Thanks, Tim, my whole Idea is, developers should be able to replay the message from the provenance for at least 5 days as per the requirements

I'm assuming the only solution is BUMP up the provenance storage to achieve replay capability.
please let me know your thoughts!.