question Re: NameNode is not leaving safemode and is not getting up. Need help. in Archives of Support Questions (Read Only)

NameNode is not leaving safemode and is not getting up. Need help.

chandrika_patur — Fri, 03 Jun 2016 19:03:49 GMT

I'm using Ambari 2.2.1.0 and HDP 2.3.4.0

Using Ambari UI, I tried to start my NN. But, its showing below error.

safemode: Call From server1.ddns.net/141.178.0.16 to server1.ddns.net:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2016-06-03 12:40:06,640 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://server1.ddns.net:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From server1.ddns.net/141.178.0.16 to server1.ddns.net:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

After that, I tried below commands and the outputs are:

[root@server1 ~]# sudo -u hdfs hdfs dfsadmin -report
report: Call From server1.ddns.net/141.178.0.16 to server1.ddns.net:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

[root@server1 ~]# sudo -u hdfs hdfs dfsadmin -safemode enter
safemode: Call From server1.ddns.net/141.178.0.16 to server1.ddns.net:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

[root@server1 ~]# sudo -u hdfs hdfs dfsadmin -safemode leave
safemode: Call From server1.ddns.net/141.178.0.16 to server1.ddns.net:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

[root@server1 ~]#

1) Why is it not leaving from safemode?

2) What could be the solution to resolve my problem and make the NN up?

Please help me in this regard....

Re: NameNode is not leaving safemode and is not getting up. Need help.

TerryP — Fri, 03 Jun 2016 20:54:54 GMT

@c pat

You need to first find out why the connection to the Namenode is being refused. Is the Namenode process up? You can do a quick check with ps aux | grep -i namenode

If the Namenode process is up, then look at the logs in /var/log/hadoop/hdfs. You will want to look at the file with that looks like hadoop-hdfs-namenode-*.log. This should help you narrow down the cause a bit.

Re: NameNode is not leaving safemode and is not getting up. Need help.

chandrika_patur — Sat, 04 Jun 2016 01:58:10 GMT

Thanks for the reply.

Below are the results of each one you asked to execute.

[root@server1 hdfs]# ps aux | grep -i namenode
root     27093  0.0  0.0 103308   900 pts/0    S+   19:40   0:00 grep -i namenode

2) [root@server1 hdfs]# nano hadoop-hdfs-namenode-server1.ddns.net.log

2016-06-03 14:03:11,218 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping NameNode metrics system...
2016-06-03 14:03:11,219 INFO  impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thre$
2016-06-03 14:03:11,219 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - NameNode metrics system stopped.
2016-06-03 14:03:11,219 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - NameNode metrics system shutdo$
2016-06-03 14:03:11,220 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.net.BindException: Port in use: server1.ddns.net:50070
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:892)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:716)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:951)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:935)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:444)
        at sun.nio.ch.Net.bind(Net.java:436)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:914)
        ... 8 more
2016-06-03 14:03:11,223 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2016-06-03 14:03:11,260 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at server1.ddns.net/141.178.0.16

In my core-site.xml, below is the entry

<property>
      <name>fs.defaultFS</name>
      <value>hdfs://server1.ddns.net:8020</value>
</property>

In my hdfs-site.xml, below are the entry

<property>
      <name>dfs.https.port</name>
      <value>50470</value>
</property>

<property>
	<name>dfs.namenode.http-address</name>
	<value>server1.ddns.net:50070</value>
</property>

<property>
	<name>dfs.namenode.https-address</name>
	<value>server1.ddns.net:50470</value>
</property>

<property>
  <name>dfs.namenode.rpc-address</name>
  <value>server1.ddns.net:8020</value>
</property>

<property>
  <name>dfs.namenode.safemode.threshold-pct</name>
  <value>1</value>
</property>

Re: NameNode is not leaving safemode and is not getting up. Need help.

jyadav — Sat, 04 Jun 2016 02:08:23 GMT

@c pat

Looks like your namenode process is down due to "Port in use: server1.ddns.net:50070".

Check if any process occupying the port 50070.

lsof -i:50070

If output shows nothing then can you please start the NN again and run the above command to see if NN process running on that port?.

Re: NameNode is not leaving safemode and is not getting up. Need help.

chandrika_patur — Sat, 04 Jun 2016 02:15:33 GMT

Even after re run the NN, nothing is getting displayed

[root@server1 conf]# lsof -i:50070
[root@server1 conf]#

Re: NameNode is not leaving safemode and is not getting up. Need help.

jyadav — Sat, 04 Jun 2016 02:20:31 GMT

Can you check latest NN logs?

Re: NameNode is not leaving safemode and is not getting up. Need help.

chandrika_patur — Sat, 04 Jun 2016 06:23:38 GMT

Any help would be appreciated...

Thanks in advance.

Re: NameNode is not leaving safemode and is not getting up. Need help.

chandrika_patur — Sun, 05 Jun 2016 07:04:31 GMT

Latest NN logs:

2016-06-05 01:03:14,805 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - NameNode metrics system stopped.
2016-06-05 01:03:14,806 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - NameNode metrics system shutdo$
2016-06-05 01:03:14,806 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.net.BindException: Port in use: server1.ddns.net:50070
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:892)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:716)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:951)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:935)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:444)
        at sun.nio.ch.Net.bind(Net.java:436)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:914)
        ... 8 more
2016-06-05 01:03:14,809 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2016-06-05 01:03:14,812 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
/************************************************************

Re: NameNode is not leaving safemode and is not getting up. Need help.

jyadav — Sun, 05 Jun 2016 17:10:14 GMT

@c pat

This seems to be an issue with Ambari, Kindly restart the Ambari process and NN host to cleanup cache. Or else you can change the port 50070 to some other available port.

Curious to know your Ambari and HDP version?

Re: NameNode is not leaving safemode and is not getting up. Need help.

chandrika_patur — Sun, 05 Jun 2016 17:24:01 GMT

Ambari 2.2.1.0 and HDP 2.3.4.0

and this issue got resolved with below steps

1) stopped ambari agent and ambari server

2) did NN format using --> hdfs namenode -format

3) provided 777 rights and changed the user to hadoop, for namenode folder resided in the below path

/bigdata/hadoop/hdfs/namenode

4) restarted the agent and server.

Re: NameNode is not leaving safemode and is not getting up. Need help.

jyadav — Sun, 05 Jun 2016 17:36:56 GMT

Not sure why NN format required here.

Re: NameNode is not leaving safemode and is not getting up. Need help.

achu_thambi — Mon, 22 May 2017 18:22:54 GMT

@c pat....I couldnt find the namenode location...

Re: NameNode is not leaving safemode and is not getting up. Need help.

erkansirin78 — Thu, 14 Sep 2017 03:47:31 GMT

I did first two step but I got the same connection refused when I tried 777 rights

Re: NameNode is not leaving safemode and is not getting up. Need help.

rajadash2006 — Mon, 23 Oct 2017 19:24:25 GMT

Please Add this below lines to hdfs-site.xml in between configuration properties and restart the hdfs.

<name>dfs.safemode.threshold.pct</name>

</property>

Re: NameNode is not leaving safemode and is not getting up. Need help.

shashankkumar_m — Wed, 04 Apr 2018 04:18:10 GMT

@PRAFUL DASH

Thanks it works for me.

Re: NameNode is not leaving safemode and is not getting up. Need help.

pradhanalok009 — Wed, 10 Oct 2018 07:25:13 GMT

Hello, I am facing the same error in my cluster, and same all above thing is reflecting, how did you solve this problem. waiting for your reply. please heal me out. @c pat