question Re: Connection failed: [Errno 111] Connection refused... in Support Questions

Connection failed: [Errno 111] Connection refused...

mike_w_wong — Sun, 18 Aug 2019 05:44:00 GMT

So, I'm attempting a 'start from scratch' install of HDP2.6.4 on GCP. After 3+ days, I was finally able to get GCP into a state where I was able to get the code installed on my instances.

Then the moment of truth as I logged into Ambari...it was a sea of RED, lit up like Christmas tree! 30+ alerts and the only services running were HDFS, Zookeeper, and Flume

Digging into the Resource Manager, there seems to be a bit of a recurring theme:

Connection failed: [ERRNO 111] Connection refused to <machine_name>:<port>

At first, I thought it was simply because I hadn't opened up those ports in the GCP firewall, so I added them. But I'm still encountering the errors.

Any ideas where I've gone wrong?

Re: Connection failed: [Errno 111] Connection refused...

Shelton — Fri, 18 May 2018 13:15:05 GMT

@Mike Wong

Yes, your major source of the problem seems to be connectivity. Again just to validate can you walk through the steps you executed or documentation you used. Members tend to ignore basic steps that are vital.

Could you also share or dig into the logs for most of the components in /var/log/{component}/*.log

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Fri, 18 May 2018 13:56:54 GMT

@Geoffrey Shelton Okot

Here's the documentation I used to prepare and install my cluster: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-installation/content/ch_Getting_Ready.html

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Fri, 18 May 2018 14:07:11 GMT

Here are some of the errors for the respective components:

Oozie-

Stack trace for the error was (for debug purposes):
--------------------------------------
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/oozie/share/lib/lib_20180518045451/oozie/jackson-databind-2.4.4.jar could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and 2 node(s) are excluded in this operation.

HiveServer2-

raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.6.4.0-91/hive/hive.tar.gz -H 'Content-Type: application/octet-stream' 'http://hdp.c.my-project-1519895027175.internal:50070/webhdfs/v1/hdp/apps/2.6.4.0-91/hive/hive.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "File /hdp/apps/2.6.4.0-91/hive/hive.tar.gz could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and 2 node(s) are excluded in this operation.\n\tat

Spark2 History Server-

raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/tmp/spark2/spark2-hdp-yarn-archive.tar.gz -H 'Content-Type: application/octet-stream' 'http://hdp.c.my-project-1519895027175.internal:50070/webhdfs/v1/hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "File /hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded in this operation.\n\tat

Zeppelin Notebook-

raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/current/zeppelin-server/interpreter/spark/dep/zeppelin-spark-dependencies_2.11-0.7.3.2.6.4.0-91.jar -H 'Content-Type: application/octet-stream' 'http://hdp.c.my-project-1519895027175.internal:50070/webhdfs/v1/apps/zeppelin/zeppelin-spark-dependencies_2.11-0.7.3.2.6.4.0-91.jar?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "File /apps/zeppelin/zeppelin-spark-dependencies_2.11-0.7.3.2.6.4.0-91.jar could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded in this operation.\n\tat

Re: Connection failed: [Errno 111] Connection refused...

sparsh_singhal — Fri, 18 May 2018 14:12:56 GMT

@Mike Wong

Please check your /etc/hosts on all nodes. And also if selinux is disabled using getenforce command.

Re: Connection failed: [Errno 111] Connection refused...

Shelton — Fri, 18 May 2018 14:41:37 GMT

@Mike Wong

Can you describe your cluster OS & node number? Is passwordless connect working?

Your "data nodes have failed and are not registered with the name node", this is network problem.

Is your DNS working can you validate?

Your hostname in /etc/hosts shouldn't be mixed cases!!! Lowercase and FQDN

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Fri, 18 May 2018 15:23:04 GMT

@Sparsh, these four lines are present in the /etc/hosts file on all four of my nodes-

35.231.154.250 hdp.c.my-project-1519895027175.internal # Added by Mike Wong
35.231.170.209 slave1.c.my-project-1519895027175.internal #Added by Mike Wong
35.231.220.224 slave2.c.my-project-1519895027175.internal #Added by Mike Wong
35.229.111.57 slave3.c.my-project-1519895027175.internal #Added by Mike Wong

getenforce resulted in Enforcing on three of my nodes. I've since disabled selinux using 'setenforce 0'. Now all four node are Permissive

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Fri, 18 May 2018 15:25:06 GMT

@Geoffrey Shelton Okot

Cluster OS = RHEL7 VMs on GCP, four nodes total. Yes, SSH is working

I can ping from each node to the other three nodes via IP address and FQDN successfully

Re: Connection failed: [Errno 111] Connection refused...

sparsh_singhal — Fri, 18 May 2018 15:42:58 GMT

@Mike Wong

I'd recommend disabling selinux and rebooting the machines.

After that look into hdfs logs make sure hdfs is up with no alerts. Try restarting hdfs. All other services would come up then.

Re: Connection failed: [Errno 111] Connection refused...

Shelton — Fri, 18 May 2018 21:21:21 GMT

@Mike Wong

Ping is not enough. I recommend checking the below settings on all the nodes in the cluster

Disable and stop firewalld

systemctl disable firewalld
systemctl stop firewalld

Status of Firewalld

systemctl status firewalld

Check DNS

This should have the correct GW

# route -n

Disable SELinux

sed -ie 's/enforcing/disabled/g' /etc/selinux/config

Then retry

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Fri, 18 May 2018 22:36:24 GMT

All of this look ok

Firewalld status-

● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

May 15 15:09:17 localhost systemd[1]: Starting firewalld - dynamic firewall daemon...
May 15 15:09:19 localhost systemd[1]: Started firewalld - dynamic firewall daemon.
May 15 21:03:24 slave3 systemd[1]: Stopping firewalld - dynamic firewall daemon...
May 15 21:03:25 slave3 systemd[1]: Stopped firewalld - dynamic firewall daemon.

route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.142.0.1      0.0.0.0         UG    100    0        0 eth0
10.142.0.1      0.0.0.0         255.255.255.255 UH    100    0        0 eth0
10.142.0.5      0.0.0.0         255.255.255.255 UH    100    0        0 eth0

Re: Connection failed: [Errno 111] Connection refused...

Shelton — Sat, 19 May 2018 00:31:14 GMT

@Mike Wong

Can you past the whole /etc/hosts entry? I am interested in the first 4 lines !

You should have some lines like

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

Please confirm

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Sat, 19 May 2018 02:37:04 GMT

@Geoffrey Shelton Okot

Whole /etc/hosts file-

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.142.0.2 hdp.c.my-project-1519895027175.internal hdp  # Added by Google
169.254.169.254 metadata.google.internal  # Added by Google
35.231.154.250 hdp.c.my-project-1519895027175.internal # Added by Mike Wong
35.231.170.209 slave1.c.my-project-1519895027175.internal #Added by Mike Wong
35.231.220.224 slave2.c.my-project-1519895027175.internal #Added by Mike Wong
35.229.111.57 slave3.c.my-project-1519895027175.internal #Added by Mike Wong

Re: Connection failed: [Errno 111] Connection refused...

Shelton — Sat, 19 May 2018 02:57:06 GMT

@Mike Wong

The entries look correct 🙂

Can you try to start YARN manually?

su -l yarn -c "/usr/hdp/current/hadoop-yarn-resourcemanager/sbin/yarn-daemon.sh start resourcemanager"

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Sat, 19 May 2018 03:32:46 GMT

Hmmm, when I try to start RM, I'm getting this-

2018-05-18 20:26:11,400 INFO  recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1227)) - Exception while executing a ZK operation.
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /rmstore
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:326)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:322)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1174)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1207)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createRootDir(ZKRMStateStore.java:336)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createRootDirRecursively(ZKRMStateStore.java:1311)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.startInternal(ZKRMStateStore.java:303)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStart(RMStateStore.java:598)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:593)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1008)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1049)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1045)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1045)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1085)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1229)
2018-05-18 20:26:11,400 INFO  recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1230)) - Retrying operation on ZK. Retry no. 203
2018-05-18 20:26:11,471 INFO  zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server slave1.c.my-project-1519895027175.internal/10.142.0.3:2181. Will not attempt to authenticate using SASL (unknown error)
2018-05-18 20:26:11,472 INFO  zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established, initiating session, client: /10.142.0.3:52748, server: slave1.c.my-project-1519895027175.internal/10.142.0.3:2181 
2018-05-18 20:26:11,472 INFO  zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

Re: Connection failed: [Errno 111] Connection refused...

Shelton — Sat, 19 May 2018 03:45:02 GMT

@Mike Wong

I hope this is not production !

1) Stop Resource Manager

2) Connect with ZK server in /usr/hdp/2.x.x/zookeeper the below uout put is from cluster your output could look slightly different depending on installed components

./bin/zkCli.sh[zk: 
localhost:2181(CONNECTED) 0] ls /
[registry, cluster, brokers, storm, zookeeper, infra-solr, hbase-unsecure, admin, isr_change_notification, templeton-hadoop, hiveserver2, controller_epoch, druid, rmstore, ambari-metrics-cluster, consumers, config]
[zk: localhost:2181(CONNECTED) 1] ls /rmstore
[ZKRMStateRoot]
[zk: localhost:2181(CONNECTED) 2]

3) Remove the znode for RM -- rmr /rmstore

[zk: localhost:2181(CONNECTED) 2] rmr /rmstore

Restart YARN

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Sun, 18 Aug 2019 05:43:46 GMT

1. I'm guessing my RM is already stopped

2. When I try to launch the Zookeeper Cli (./bin/zkCli.sh), I'm getting the following-

Connecting to localhost:2181
2018-05-18 23:18:06,066 - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-91--1, built on 01/04/2018 10:34 GMT
2018-05-18 23:18:06,068 - INFO  [main:Environment@100] - Client environment:host.name=slave1.c.my-project-1519895027175.internal
2018-05-18 23:18:06,068 - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_112
2018-05-18 23:18:06,070 - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2018-05-18 23:18:06,070 - INFO  [main:Environment@100] - Client environment:java.home=/usr/jdk64/jdk1.8.0_112/jre
2018-05-18 23:18:06,070 - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/hdp/2.6.4.0-91/zookeeper/bin/.......
.
.
.

2018-05-18 23:18:06,094 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Welcome to ZooKeeper!
.
.
.

2018-05-18 23:18:10,760 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-05-18 23:18:10,761 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@864] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:33932, server: localhost/0:0:0:0:0:0:0:1:2181
2018-05-18 23:18:10,761 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1142] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

Re: Connection failed: [Errno 111] Connection refused...

Shelton — Sat, 19 May 2018 13:49:30 GMT

@Mike Wong

YES the status shows stopped. Did you do the previous steps and restart the RM manually?

Re: Connection failed: [Errno 111] Connection refused...

mike_w_wong — Mon, 21 May 2018 10:17:45 GMT

@Geoffrey Shelton Okot

I tried to connect to the Zookeeper server, but I'm getting the above error. (closing socket connection...)

Re: Connection failed: [Errno 111] Connection refused...

andrew_ryan1 — Mon, 29 Oct 2018 07:17:16 GMT

@Mike Wong

I had an issue similar when I enabled SSL on HDFS. Map Reduce and YARN.

Symptoms:

I could connect to HDFS rest interface from the command line internal to the cluster using curl
I could run the UI for HDFS from internal to the cluster using xwindows enabled putty an XMing
OpenSSL form internal returned 0 for the HDFS connection
No external connection to the could be made using curl/openssl/telnet to HDFS
The ResourceManager and JobHistory UI's both worked fine
We had a firewall but even when disabled the connection to HDFS failed

The issue was that we have 'internal network IP addresses (infiniband)' and externally accessible IP addresses. The HDFS when enabling https had bound to the internal address and wasn't listening on the external address - hence the connection refused.

The solution was to add the

dfs.namenode.https-bind-host=0.0.0.0 property to get the service to listen across all network interfaces.

Might be worth checking if you (or anybody else that gets a connection refused errors) that have multiple network interfaces, to which the port is binding to.

netstat -nlp | grep 50470

tcp　　　　　　　 0　　　　　 0 <internal IP address>:50470　　　　　　　　0.0.0.0:*　　　　　　　　　　　　　　 LISTEN　　　　　 23521/java

netstat -nlp | grep 50070

tcp　　　　　　　 0　　　　　 0 0.0.0.0:50070　　　　　　　　　　 0.0.0.0:*　　　　　　　　　　　　　　 LISTEN　　　　　 23521/java