Is there any problem with the JVM of NiFi, is there any failover/ practice

dhieru — Tue, 24 Oct 2017 04:14:29 GMT

Hi All,

Thanks a lot to this community.

I am working on building data flows, and i have 40 to 50 data flows to be configured. some of them are like 15000 events per second.

If I have some of the data flows configured and then a very high event per second data source is being configured, and some how fills up the JVM, everything crashes because of the disk space,

or sudden event burst occurs

Does it mean we will have to proactively watch for any space issues such as JVM heap size, content repo and provenance repo. etc

Thanks

Dheiru

Re: Is there any problem with the JVM of NiFi, is there any failover/ practice

ahadjidj — Tue, 24 Oct 2017 11:44:26 GMT

Hi @dhieru singh

You need to do two things:

First, you need a good capacity planing to evaluate the required infrastructure that can handle your data flows. Consider the worst case scenario to have room for improvement and the capacity to manage bursts. There are several resources out there that can help you

https://community.hortonworks.com/articles/135337/nifi-sizing-guide-deployment-best-practices.html

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_command-line-installation/content/hdf_isg_hardware.html

Second, as you said, you need to monitor your system at different level. Pierre has a set of articles on this topic that I recommend you to read : https://pierrevillard.com/2017/05/11/monitoring-nifi-introduction/

question Is there any problem with the JVM of NiFi, is there any failover/ practice in Archives of Support Questions (Read Only)

Is there any problem with the JVM of NiFi, is there any failover/ practice

Re: Is there any problem with the JVM of NiFi, is there any failover/ practice