@srinivas p
Based on the following GC logging
Detected pause in JVM or host machine (eg GC): pause of approximately 23681msGC pool 'ParNew' had collection(s): count=1 time=0msGC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=23769ms
We see that the GC pause is comparatively very high (around 23 seconds) which can happen if the GC is not happening very aggressively, so the heap keeps growing with the time until it reaches to 90+% of the whole DataNode heap then the GC gets triggered.
In this case we can make the GC to happen nit aggressively by adding the following options in the DataNode JVM settings "HADOOP_DATANODE_OPTS"
Ambari UI --> HDFS --> Configs (Tab) --> Advanced (child Tab)--> "hadoop-env template" -> Find all the "HADOOP_DATANODE_OPTS" (includinf if .. else) both the blocks and add the following settings.
-XX:CMSInitiatingOccupancyFraction=60 -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction : The Throughput Collector starts a GC cycle only when the heap is full, i.e., when there is not enough space available to store a newly allocated or promoted object. With the CMS Collector, it is not advisable to wait this long because it the application keeps on running (and allocating objects) during concurrent GC. Thus, in order to finish a GC cycle before the application runs out of memory, the CMS Collector needs to start a GC cycle much earlier than the Throughput Collector.
-XX+UseCMSInitiatingOccupancyOnly : We can use the flag -XX+UseCMSInitiatingOccupancyOnly to instruct the JVM not to base its decision when to start a CMS cycle on run time statistics. Instead, when this flag is enabled, the JVM uses the value of CMSInitiatingOccupancyFraction for every CMS cycle, not just for the first one. However, keep in mind that in the majority of cases the JVM does a better job of making GC decisions than us humans. Therefore, we should use this flag only if we have good reason (i.e., measurements) as well as really good knowledge of the lifecycle of objects generated by the application.
Also Recommendation is to set the young generation heap size (-XX:MaxNewSize) to be set to (1/8th) of total Max heap. Young gen has the parallel collectors and most short lived objects are removed before it promoting to old gen, same old gen has enough space and threads to handle the promotions.
Also can you please check your filesystem to see if the DataNode process is crashing and generating the "hs_err_pid" files? If due to some reason the DataNode JVM process is getting crashed then you should see a file "hs_err_pid" it can be helpful to understand why the DataNode process is getting crashed. Your DataNode process might have the following option enabled by default which tells the JVM to generate the crash dump text file in the following location (just in case if it is crashing).
-XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log
.
