Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issue : "sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode

Highlighted

Issue : "sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode

Hi,

 

In my CDA, on HDP while system is starting it is throwing below error:

"sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode. Below is the more detailed log.

Could you please let me know what should be done since it is not starting namenode and hence not letting to start entire system.

stderr: 

 stdout:
2019-07-20 01:20:47,602 - Waiting for this NameNode to leave Safemode due to the following conditions: HA: False, isActive: True, upgradeType: None
2019-07-20 01:20:47,604 - Waiting up to 19 minutes for the NameNode to leave Safemode...
2019-07-20 01:20:47,604 - Execute['/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF''] {'logoutput': True, 'tries': 115, 'user': 'hdfs', 'try_sleep': 10}
safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.3 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2019-07-20 01:20:51,904 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.3 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2019-07-20 01:21:06,242 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2019-07-20 01:21:19,389 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2019-07-20 01:21:31,853 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2019-07-20 01:21:45,871 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 
2019-07-20 01:21:57,932 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
5 REPLIES 5

Re: Issue : "sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode

Please note I am using HDP 3.0 version

Re: Issue : "sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode

Hi Development Team,

 

Could you please let me know if any one looked at this issue,

Re: Issue : "sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode

Super Mentor

@shashank_naresh 

The following error can occur if the NameNode is not running fine.

Highlighted Error:

 

Call From sandbox-hdp.hortonworks.com/172.18.0.3 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  <a href="http://wiki.apache.org/hadoop/ConnectionRefused" target="_blank">http://wiki.apache.org/hadoop/ConnectionRefused</a>

 

Possible cause:
1). Please verify if the port 8020 is actually listening?  If not then we will need to check the NameNode log for any errors. We might need to also check the NameNode GC log to see if it has sufficient memory and if the GC is happening properly or not?

 

# netstat -tnlpa | grep 8020
# netcat -tnlpa | grep 50070

 

2). There might be some errors listed in NameNode logs that can be found at. So can you please check and share the log file here?

 

# ls -l  /var/log/hadoop/hdfs/hadoop-hdfs-namenode-sandbox-hdp.hortonworks.com.log
# ls -lart /var/log/hadoop/hdfs/gc.log-201908*

 

Specially in case of Sandbox Environment:

As Sandbox is a single node cluster for testing/learning purpose which has a lots of services running on a single host so some times we might see that due to heavy load of other running services other services like NameNode ..etc does not function properly.  

So please try this:  Stop the services that you are not needing currently in your Sandbox. Put those services in maintenance mode form Ambari UI and then just start the services that you are testing currently.  This should free some memory on the sandbox host and back ground processing load by those services and that should improve the scenario. 

 

Re: Issue : "sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode

Cloudera Employee

Hi,

This can be caused by several reasons, we are aware that the HDP 3.0.1 Sandbox with CDA enabled requires a high amount of RAM, to this out as one of the causes, could you please tell us how much RAM being used? Further please include which Sandbox flavor you are using (VMWare, VBox, Docker).

Thank you.

Re: Issue : "sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.connectexception: connection refused;" for namenode

Hi @cfarnes 

 

Thanks for reply, I gave 28GB RAM to Virtual box to run CDA.