https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97360#M60578 <P> When we run HDF on a single machine , does all the data flow build on that machine run under a single JVM?</P><P>I did see in Nifi documents which talks about how how you can control the spill the data from JVM to hardisk. But is there option to run via multiple JVM say one for each flow. Also How big of a JVM size you usually have for a edge node. </P> Fri, 20 Nov 2015 06:46:40 GMT csankaraiah 2015-11-20T06:46:40Z https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97360#M60578 <P> When we run HDF on a single machine , does all the data flow build on that machine run under a single JVM?</P><P>I did see in Nifi documents which talks about how how you can control the spill the data from JVM to hardisk. But is there option to run via multiple JVM say one for each flow. Also How big of a JVM size you usually have for a edge node. </P> Fri, 20 Nov 2015 06:46:40 GMT https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97360#M60578 csankaraiah 2015-11-20T06:46:40Z https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97361#M60579 <P>When you run HDF on a single machine it is a single JVM process. That instance has three internal repositories (content, flow file, and provenance) which are what can be controlled in the configuration as to how much of the repositories is retained on disk. For high performance it is best to have each of the repositories using a separate disk. </P><P>One instance can have many logical flows which can be optionally grouped inside processors groups. There can be many disconnected logical flows with in one instance. There are discussions for future capabilities where logical flows could be restricted to only certain groups of users.</P><P>The default for memory requirements for NiFi out-of-the-box are 512MB, so at the moment that would probably be the starting point for an edge node.</P> Fri, 20 Nov 2015 07:27:19 GMT https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97361#M60579 bbende 2015-11-20T07:27:19Z https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97362#M60580 <P>Thanks <A rel="user" href="https://community.cloudera.com/users/363/bbende.html" nodeid="363">@bbende</A>!! So we do not recommend scaling Nifi vertical by increasing the heap size for the JVM to really large size? </P> Fri, 20 Nov 2015 09:09:00 GMT https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97362#M60580 csankaraiah 2015-11-20T09:09:00Z https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97363#M60581 <P>We definitely do recommend increasing the heap appropriately for the given use case. I was focused on some of the other aspects and forgot to mention that <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P><P>In conf/bootstrap.conf there are the following default settings which can be increased:</P><P># JVM memory settings </P><P>java.arg.2=-Xms512m </P><P>java.arg.3=-Xmx512m</P><P>It is hard to recommend a general heap size for all use cases, but anywhere from 512MB up to 8GB is common.</P> Fri, 20 Nov 2015 09:32:36 GMT https://community.cloudera.com/t5/Support-Questions/How-to-scale-vertically-for-a-HDF-instance/m-p/97363#M60581 bbende 2015-11-20T09:32:36Z