what is the recommendation for Datanode heap size and new generation heap size?
now i set datanode heapsize to 24 GB and new genreration heap size to 10 GB.
2017-07-15T13:50:33.501-0500: 4009.839: [Full GC (Allocation Failure) 2017-07-15T13:50:33.501-0500: 4009.840: [CMS2017-07-15T13:50:39.567-0500: 4015.905: [CMS-concurrent-mark: 12.833/12.841 secs] [Times: user=20.33 sys=5.59, real=12.84 secs] (concurrent mode failure): 14680064K->14680064K(14680064K), 39.2851287 secs] 24117247K->22948902K(24117248K), [Metaspace: 36771K->36771K(1083392K)], 39.2852865 secs] [Times: user=39.18 sys=0.04, real=39.29 secs] . 2017-07-15T13:52:15.250-0500: 4111.588: [Full GC (Allocation Failure) 2017-07-15T13:52:15.250-0500: 4111.588: [CMS2017-07-15T13:52:21.412-0500: 4117.750: [CMS-concurrent-mark: 12.025/12.030 secs] [Times: user=17.74 sys=1.38, real=12.03 secs] (concurrent mode failure): 14680063K->14680063K(14680064K), 39.5266803 secs] 24117247K->23076661K(24117248K), [Metaspace: 36781K->36781K(1083392K)], 39.5268469 secs] [Times: user=39.41 sys=0.05, real=39.53 secs]
We see that out of 24GB almost all 24GB is being utilized by the DataNode and the Garbage collector is hardly able to clean up the 1 GB memory.
24117247K->22948902K(24117248K) AND 24117247K->23076661K(24117248K)
It indicates that the Heap Size is not sufficient for the DataNode or the DataNode cache settings are not appropriately set.
- Can you please share the core-site.xml and hdfs-site.xml
- Some issues are reported for similar behavior: https://issues.apache.org/jira/browse/HDFS-11047
We have a cluster running HDP 2.5 with 3 worker nodes and around 9.1 million blocks with an average block size of 0.5 MB.Is could be the reason for frequent JVM pause ?
Does all your datanode is affected by this issues / SIngle DataNode is only affected. Have u tried to re-balance HDFS to see if issue is fixed.