HDFS NameNode won't leave safemode

[#1]

Hello,

- I've setup a Sandbox HDP 3.0.1.
- Determine Network Adapter of Your VirtualBox Sandbox (Bridged Adapter, IP:192.168.1.37).
- I edited the hosts file (on my computer, windows) and assigned the domain sandbox-hdp.hortonworks.com to the IP address of 192.168.1.37, but When I start all services via the Ambari UI, I receive the following error message:

safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.2 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

2019-09-29 18:25:10,968 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.2 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused.

I do not know how to fix the error. Who knows, please just help, thank you!

[#2]

@ThanhP 

I can see you are accessing the web shell UI on 192.168.1.37 yet the sandbox is pointing to 172.18.0.2:8020. Now if you type in your browser http://192.168.1.37:1080/splash2.html you should land on the splash screen
Choose the Ambari UI link , this should give you the normal Ambari user UI if you reset your password before as I had stated please use that admin/[password] combination and you will realize that none of the components are started HDFS, YARN, MR, HIVE, HBASE etc Note: some components are in maintenance mode unless you want any of them beware of the dependencies e.g ATLAS must have HBASE running etc.. that could explain why you got the error "Failed on connection error java.net.Connection"

At a certain point during your startup, "Timeline Service V1.5" will error out with the message to do with "safemode on" get to the shell as hdfs

$ hdfs dfsadmin -safemode get
This should indicate it's ON so get it out of safemode

$ hdfs dfsadmin -safemode leave

And again use the option Ambari start all it will pick up from the last point of failure, after the components startup you can simply access them eg the name node UI will automatically point to http://sandbox-hdp.hortonworks.com:50070/dfshealth.html#tab-overview

Please do that an revert

[#3]

Hi @Shelton 

Before, reading your instructions, I tried running this command

sudo -u hdfs hdfs namenode -format

After executing the above command, NAMENODE has run

Thank you for your enthusiastic help

[#4]

@ThanhP 

Good everything is perfect for you now 🙂

You ONLY execute sudo -u hdfs hdfs namenode -format  as a last resort because it's dangerous and not recommended to run that on production cluster as that [re-initializes] formats your Namenode hence deleting all your metadata stored on the NameNode.

Having said that the answer you accepted can't help a member who encounters the same issue "HDFS NameNode won't leave safemode" maybe you should un-accept it and accept your own answer as it's the more realistic answer .
Happy hadooping 

[#5]

@Shelton 

Thanks to your reminder, I will learn from experience so as not to affect the community. Thanks for your help!

[#6]

@Shelton 

I could not start namenode with ambari ui when setting safemode to off mode.

Please could you helpme out!

And also while setting safemode to 'ON'  does not allow me to do task.