question Forcefully Exit NameNode Safe Mode in Archives of Support Questions (Read Only)

Forcefully Exit NameNode Safe Mode

andisonde — Fri, 09 Mar 2018 11:22:18 GMT

I restored my HDP 2.6.4 cluster from instance snapshots, restarted and most of services started successfully. After a quick investigation found out that NN is in safe mode. Ok. Let's take it out of that mode with:

hdfs dfsadmin -safemode leave

Not so quick, it the command execution would report:

Safe mode is OFF.

Let me search for a forceful exit. I could not find the command. Please help.

Re: Forcefully Exit NameNode Safe Mode

cstanca — Fri, 09 Mar 2018 11:29:42 GMT

@Andi Sonde

Most likely, Name Node detected blocks with generation stamps in future. It happened to me on a similar restore. Your snapshots were not consistent because they were "hot". All services and then all server nodes should have been stopped before taking a consistent snapshot ("cold"). This means that Name Node metadata is inconsistent. Exiting safe mode could cause loss of data. Please restart name node with right metadata if you have it somewhere or use "hdfs dfsadmin -safemode forceExit", if you are certain that the NameNode was started with the correct FsImage and edit logs. If you encountered this during a rollback, it is safe to exit with -safemode forceExit."

Re: Forcefully Exit NameNode Safe Mode

andisonde — Fri, 09 Mar 2018 11:34:51 GMT

@Constantin Stanca

Command worked. I have some bad blocks, but it is not a big deal. This is my sandbox environment. Lesson learned to take a cold backup. Thank you so much!