Support Questions

Find answers, ask questions, and share your expertise

secondary NN failing to checkpoint


The filesystem that is used for the secondary namenode filled up. I cleaned up the filesystem, but since then it has failed to do any checkpoints. It looks like its trying over and over again and failing, but I don't know why. Here are the logs:

2017-07-31 16:23:34,711 ERROR namenode.SecondaryNameNode ( - Exception in doCheckpoint Unable to download to any storage directory at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.receiveFile( at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl( at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient( at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.downloadEditsToStorage( at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$ at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$ at Method) at at at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles( at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint( at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork( at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$ at at at



@Alex Eifler

You can manually force a check point see doc . Is it linked to the secondary namenode's local storage problem. Can you check value of dfs.namenode.checkpoint.dir and see if any issues like RO mount or storage full or bad disk maybe?

As the hdfs user.

hadoop secondarynamenode -checkpoint force


Check your mounted disk status whether or not to write on disk.