Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Forcefully Exit NameNode Safe Mode

avatar
Contributor

I restored my HDP 2.6.4 cluster from instance snapshots, restarted and most of services started successfully. After a quick investigation found out that NN is in safe mode. Ok. Let's take it out of that mode with:

hdfs dfsadmin -safemode leave

Not so quick, it the command execution would report:

Safe mode is OFF.

Let me search for a forceful exit. I could not find the command. Please help.

1 ACCEPTED SOLUTION

avatar
Super Guru

@Andi Sonde

Most likely, Name Node detected blocks with generation stamps in future. It happened to me on a similar restore. Your snapshots were not consistent because they were "hot". All services and then all server nodes should have been stopped before taking a consistent snapshot ("cold"). This means that Name Node metadata is inconsistent. Exiting safe mode could cause loss of data. Please restart name node with right metadata if you have it somewhere or use "hdfs dfsadmin -safemode forceExit", if you are certain that the NameNode was started with the correct FsImage and edit logs. If you encountered this during a rollback, it is safe to exit with -safemode forceExit."

View solution in original post

2 REPLIES 2

avatar
Super Guru

@Andi Sonde

Most likely, Name Node detected blocks with generation stamps in future. It happened to me on a similar restore. Your snapshots were not consistent because they were "hot". All services and then all server nodes should have been stopped before taking a consistent snapshot ("cold"). This means that Name Node metadata is inconsistent. Exiting safe mode could cause loss of data. Please restart name node with right metadata if you have it somewhere or use "hdfs dfsadmin -safemode forceExit", if you are certain that the NameNode was started with the correct FsImage and edit logs. If you encountered this during a rollback, it is safe to exit with -safemode forceExit."

avatar
Contributor

@Constantin Stanca

Command worked. I have some bad blocks, but it is not a big deal. This is my sandbox environment. Lesson learned to take a cold backup. Thank you so much!