Restarting Nifi due to RAM and Disk space issue once in a day

[#1]

According to our current implementation, we are restarting nifi once in a day becuase after lot of porcessing, either disk space or RAM is fully cosumed by Nifi. That is why we need to restart the nifi and we are not only restarting the nifi we are cleaning the below directories also -

1. content_repository
2. state
3. logs
4. run
5. work
6. database_repository
7. provenance_repository
8. flowfile_repository
   

I know here is something wrong in our configuration and setup which is forcing us to restarting nifi once in a day and the above folder cleaning. Please help if you find something wrong in our process.

[#2]

@Bryan Bende - I request you to please give your thoughts on it. Thanks in advance.

[#3]

You can turn down the length of provenance data or change the repository type.

Seems like you need more RAM and more nodes.

Post your configuration files.

How many flows?

What version of NiFi?

What version of JDK?

What host OS?

How much RAM on the system?

What kind of server.

What type of disk?

Configuration Recommendations:

https://community.hortonworks.com/articles/135337/nifi-sizing-guide-deployment-best-practices.html

I would double all of those numbers.

Guides:

Set rollovers shorter

Upgrade to the latest HDF 3.3 / NiFi 1.8.0 version.

References:

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices

https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.h...

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/apache-nifi-overview/content/what-is-apache...

[#4]

Thanks for the response.

Configurations -

How many flows? - did you mean messages in nifi? if yes, almost 45k messages flows as we are using splits and all. you can say almost 1.2 GB of data flows.

What version of NiFi? - 1.7.1

What version of JDK? - 1.8

What host OS? - Redhat

How much RAM on the system? System is having 32 Gb of RAM. 6GB is allocated to Nifi in bootstrap.conf.

What kind of server? -

What type of disk? - HDD