New Contributor
Posts: 1
Registered: ‎07-25-2016

Cloudera Express 5.7.0 on Virtual Machine Resource Allocation



First of all, I am new to cloudera and big data solutions and I am trying to set up a training lab.

I am using a virtual machine on an ESXi server ( hypervisor) which has 24GB of RAM allocated to it.


I have downloaded the vm from the cloudera site : Cloudera Express 5.7.0 Vanilla and deployed it succesfully.


The problems I am facing right now are about Java Heap allocation. I saw that the manager automatically assgined values to Java Heap sizes for the NameNode and Secondary NameNode which are not enough ( I don't understand why, because the machine has 24GB and I saw that 8+ should be enough ).


I manually altered those values to 4GB for each node, but I got another error regarding space issues - Memory on host quickstart.cloudera is overcommitted. The total memory allocation is 21.3 GiB bytes but there are only 23.5 GiB bytes of RAM (4.7 GiB bytes of which are reserved for the system). Visit the Resources tab on the Host page for allocation details. Reconfigure the roles on the host to lower the overall memory allocation. Note: Java maximum heap sizes are multiplied by 1.3 to approximate JVM overhead


So basically, if I increase the Heap for the NameNodes to the suggested size, I get the above error, if I keep it lower, then I get errors on not having enough...


My questions would be:

1. Is 24 GB enough for the current version of virtual machine I am using ? - it has only one data node and 1 name node + secondary node, as it is a training machine I espect low processing on it.

2. If 24 is not ok, what size do you recommend ? - I think we can pump it up to 32.

3. What settings should I use for the heap allocation, can I keep the defaults ones or should I increase them to 4GB ( this is what the tool is suggesting ) or some other value ?

4. Any other tips?


Thanks in advance for your answers,