Secondary NN Heap Requirements.

[#1]

Hi All,

I would like to know what is heap value that I need to set for SNN ?

I have a big cluster with ~600 nodes and ~150M blocks so I had set NN heap to be 150GB but what value should I keep for SNN ?

Currently its running with 8GB, I feel its low as it needs more MEM to perform checkpointing. 

Each hosts are running with 750GB of MEM.

Please suggest.

[#2 · Accepted solution]

Hi @kolli_sandeep , I believe you are talking about Secondary Namenode and not the Standby Namenode on a Non-HA cluster. If so, the functionality of the Secondary Namenode would be to perform the checkpointing of the fsimage with the new edits.

The Secondary NameNode does require some memory to hold the data structures and objects necessary for merging the edits log and the FsImage. However, it does not load the entire FsImage into memory at once. As such, I would suggest keeping the heap to atleast 2x times the fsimage fpr smooth functioning.