@Amn_468 This is due to the Java Heap Size. Let's say the default setting for the namenode_java_heapsize is 1GB. Cloudera recommends having 1GB of heap space for every 1M blocks in a cluster. If the data in your cluster is growing rapidly, factor in the potential future number of blocks your cluster will require when determining the size setting, so you can avoid having to restart the namenode. It is only possible to change the setting by restarting the namenode. Calculating the Required Heap Size Determine the number of blocks in the cluster. This information is available on the namenode web UI under the Summary section, with information like the following: 117,387 files and directories, 56,875 blocks = 174,262 total filesystem object(s). Alternatively, the information is available from the output of the fsck command: Total size: 9958827546 B (Total open files size: 93 B)
Total dirs: 20397
Total files: 57993
Total symlinks: 0 (Files currently being written: 1)
Total blocks (validated): 56874 (avg. block size 175103 B) (Total open file blocks (not validated): 1)
... Given the number of blocks, allocate 1GB of heap space for each 1M blocks, plus some additional memory for growth. For example, if there are 6,543,567 blocks, you need 6.5GB of heap to cover the current cluster size, but 8GB would be a sensible setting to allow for growth of the cluster. After that you can adjust the Java Heap Size for NN. Hope this helps.
... View more
