Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Restarting Nifi due to RAM and Disk space issue once in a day

Restarting Nifi due to RAM and Disk space issue once in a day

Explorer

According to our current implementation, we are restarting nifi once in a day becuase after lot of porcessing, either disk space or RAM is fully cosumed by Nifi. That is why we need to restart the nifi and we are not only restarting the nifi we are cleaning the below directories also -

  1. content_repository
  2. state
  3. logs
  4. run
  5. work
  6. database_repository
  7. provenance_repository
  8. flowfile_repository

I know here is something wrong in our configuration and setup which is forcing us to restarting nifi once in a day and the above folder cleaning. Please help if you find something wrong in our process.

3 REPLIES 3
Highlighted

Re: Restarting Nifi due to RAM and Disk space issue once in a day

Explorer

@Bryan Bende - I request you to please give your thoughts on it. Thanks in advance.

Highlighted

Re: Restarting Nifi due to RAM and Disk space issue once in a day

Super Guru

You can turn down the length of provenance data or change the repository type.

Seems like you need more RAM and more nodes.

Post your configuration files.

How many flows?

What version of NiFi?

What version of JDK?

What host OS?

How much RAM on the system?

What kind of server.

What type of disk?

Configuration Recommendations:

https://community.hortonworks.com/articles/135337/nifi-sizing-guide-deployment-best-practices.html

I would double all of those numbers.

Guides:

Set rollovers shorter

Upgrade to the latest HDF 3.3 / NiFi 1.8.0 version.

References:

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices

https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.h...

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/apache-nifi-overview/content/what-is-apache...

Highlighted

Re: Restarting Nifi due to RAM and Disk space issue once in a day

Explorer

Thanks for the response.

Configurations -

How many flows? - did you mean messages in nifi? if yes, almost 45k messages flows as we are using splits and all. you can say almost 1.2 GB of data flows.

What version of NiFi? - 1.7.1

What version of JDK? - 1.8

What host OS? - Redhat

How much RAM on the system? System is having 32 Gb of RAM. 6GB is allocated to Nifi in bootstrap.conf.

What kind of server? -

What type of disk? - HDD

Don't have an account?
Coming from Hortonworks? Activate your account here