Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Forcefully Exit NameNode Safe Mode

Solved Go to solution

Forcefully Exit NameNode Safe Mode

New Contributor

I restored my HDP 2.6.4 cluster from instance snapshots, restarted and most of services started successfully. After a quick investigation found out that NN is in safe mode. Ok. Let's take it out of that mode with:

hdfs dfsadmin -safemode leave

Not so quick, it the command execution would report:

Safe mode is OFF.

Let me search for a forceful exit. I could not find the command. Please help.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Forcefully Exit NameNode Safe Mode

@Andi Sonde

Most likely, Name Node detected blocks with generation stamps in future. It happened to me on a similar restore. Your snapshots were not consistent because they were "hot". All services and then all server nodes should have been stopped before taking a consistent snapshot ("cold"). This means that Name Node metadata is inconsistent. Exiting safe mode could cause loss of data. Please restart name node with right metadata if you have it somewhere or use "hdfs dfsadmin -safemode forceExit", if you are certain that the NameNode was started with the correct FsImage and edit logs. If you encountered this during a rollback, it is safe to exit with -safemode forceExit."

2 REPLIES 2

Re: Forcefully Exit NameNode Safe Mode

@Andi Sonde

Most likely, Name Node detected blocks with generation stamps in future. It happened to me on a similar restore. Your snapshots were not consistent because they were "hot". All services and then all server nodes should have been stopped before taking a consistent snapshot ("cold"). This means that Name Node metadata is inconsistent. Exiting safe mode could cause loss of data. Please restart name node with right metadata if you have it somewhere or use "hdfs dfsadmin -safemode forceExit", if you are certain that the NameNode was started with the correct FsImage and edit logs. If you encountered this during a rollback, it is safe to exit with -safemode forceExit."

Re: Forcefully Exit NameNode Safe Mode

New Contributor

@Constantin Stanca

Command worked. I have some bad blocks, but it is not a big deal. This is my sandbox environment. Lesson learned to take a cold backup. Thank you so much!