Issue : "sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode

[#1]

Hi,

In my CDA, on HDP while system is starting it is throwing below error:

"sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode. Below is the more detailed log.

Could you please let me know what should be done since it is not starting namenode and hence not letting to start entire system.

stderr: 

 stdout:
2019-07-20 01:20:47,602 - Waiting for this NameNode to leave Safemode due to the following conditions: HA: False, isActive: True, upgradeType: None
2019-07-20 01:20:47,604 - Waiting up to 19 minutes for the NameNode to leave Safemode...
2019-07-20 01:20:47,604 - Execute['/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF''] {'logoutput': True, 'tries': 115, 'user': 'hdfs', 'try_sleep': 10}
safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.3 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2019-07-20 01:20:51,904 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.3 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2019-07-20 01:21:06,242 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2019-07-20 01:21:19,389 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2019-07-20 01:21:31,853 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2019-07-20 01:21:45,871 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2019-07-20 01:21:57,932 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.

[#2]

Please note I am using HDP 3.0 version

[#3]

Hi Development Team,

Could you please let me know if any one looked at this issue,

[#4]

@shashank_naresh 

The following error can occur if the NameNode is not running fine.

Highlighted Error:

Call From sandbox-hdp.hortonworks.com/172.18.0.3 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  <a href="http://wiki.apache.org/hadoop/ConnectionRefused" target="_blank">http://wiki.apache.org/hadoop/ConnectionRefused</a>

Possible cause:
1). Please verify if the port 8020 is actually listening?  If not then we will need to check the NameNode log for any errors. We might need to also check the NameNode GC log to see if it has sufficient memory and if the GC is happening properly or not?

# netstat -tnlpa | grep 8020
# netcat -tnlpa | grep 50070

2). There might be some errors listed in NameNode logs that can be found at. So can you please check and share the log file here?

# ls -l  /var/log/hadoop/hdfs/hadoop-hdfs-namenode-sandbox-hdp.hortonworks.com.log
# ls -lart /var/log/hadoop/hdfs/gc.log-201908*

Specially in case of Sandbox Environment:

As Sandbox is a single node cluster for testing/learning purpose which has a lots of services running on a single host so some times we might see that due to heavy load of other running services other services like NameNode ..etc does not function properly.  

So please try this:  Stop the services that you are not needing currently in your Sandbox. Put those services in maintenance mode form Ambari UI and then just start the services that you are testing currently.  This should free some memory on the sandbox host and back ground processing load by those services and that should improve the scenario. 

[#5]

Hi,

This can be caused by several reasons, we are aware that the HDP 3.0.1 Sandbox with CDA enabled requires a high amount of RAM, to this out as one of the causes, could you please tell us how much RAM being used? Further please include which Sandbox flavor you are using (VMWare, VBox, Docker).

Thank you.

[#6]

Hi @cfarnes 

Thanks for reply, I gave 28GB RAM to Virtual box to run CDA.