Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎10-21-2017

Hadoop Secondary namenode got stuck while loading fsimage INodes

[ Edited ]

Starting secondary namenode gets stuck while loading Inodes. There are lot of GCs happening on secondary namenode, and so its not moving ahead.

 

GC is taking around 99.9% of time. Below is the message on secondary namenode health page:

"Average time spent in garbage collection was 59.9 second(s) (99.92%) per minute over the previous 5 minute(s). Critical threshold: 60.00%."

 

fsimage size: 27GB, secondary namenode heap: 100GB, GC: CMS

 

Tried following changes, but still getting similar GC results(issues).

  • Tried changing heap to 32GB,
  • Tried changing GC from CMS to G1GC

Below are the logs and it gets stuck at last statement:

tstamp_01 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://namenode:50070/imagetransfer?getedit=1&startTxId=27341007631&endTxId=27343051701&storageInfo=-60:1944437056:1471372003013:CID-03c3d692-7c29-4184-a7b8-163e61f60112
tstamp_02 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 5.77s at 42579.58 KB/s
tstamp_03 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file edits_tmp_0000000027341007631-0000000027343051701_0000000040184262756 size 0 bytes.
tstamp_04 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://namenode:50070/imagetransfer?getedit=1&startTxId=27343051702&endTxId=27343328012&storageInfo=-60:1944437056:1471372003013:CID-03c3d692-7c29-4184-a7b8-163e61f60112
tstamp_05 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.61s at 50920.92 KB/s
tstamp_06 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file edits_tmp_0000000027343051702-0000000027343328012_0000000040184268530 size 0 bytes.
tstamp_06 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode: Loading 309938509 INodes.

Any help on this will be really appreciated.

 

Announcements