Support Questions
Find answers, ask questions, and share your expertise

Hbase Regions Server getting down due to connection refused error

Contributor

new-text-document.txtHi All,

My HBase Region server is getting down due to the Connection Refused error.

I got ending with error: "Connection refused when trying to connect to '/var/lib/hadoop-hdfs/dn_socket' " and find the attached logs for reference.

What kind of error is this, how to resolve this issue.

Kindly suggest

Regards,

Mathivanan

5 REPLIES 5

Re: Hbase Regions Server getting down due to connection refused error

Mentor

@Mathi Murugan

What are the files permissions ? note the set "suid" permissions bit on dn_socket

# ls -al /var/lib/hadoop-hdfs/
total 12
drwxr-x--x   3 hdfs hadoop 4096 Oct 31 20:07 .
drwxr-xr-x. 79 root root   4096 Nov  1 03:16 ..
drwxrwxrwt   2 hdfs hadoop 4096 Jul 30 23:55 cache
srw-rw-rw-   1 hdfs hadoop    0 Oct 31 20:07 dn_socket

HDFS--> configs -->Advanced hdfs site parameter dfs.domain.socket.path should point to /var/lib/hadoop-hdfs/dn_socket

Re: Hbase Regions Server getting down due to connection refused error

Contributor

@Geoffrey Shelton Okot

As you mentioned All permission under /var/lib/hadoop-hdfs/ and HDFS--> configs -->Advanced hdfs site parameter dfs.domain.socket.path is pointing to /var/lib/hadoop-hdfs/dn_socket, are correct by default. Is there any other settings i need to change and any other ways to fix it.

Regards,

Mathivanan

Re: Hbase Regions Server getting down due to connection refused error

Mentor

@Mathi Murugan

Your regionserver could be looking for HbaseMaster but you do not have any such entry in your /etc/hosts file.

Please make sure your configuration is proper. It can be caused by loopback in your /etc/hosts files 127.0.01 hostname.

Can you share your /etc/hosts entry? If your entry looks like this:

127.0.0.1 localhost 
192.168.0.20 your.hbase.com hbase 

You have to change it like this by removing localhost:

# 127.0.0.1 localhost # line commented out 
192.166.66.66 192.168.0.20 your.hbase.com hbase localhost   # note: localhost added here

Hope that helps

Re: Hbase Regions Server getting down due to connection refused error

Contributor

hi Geoffrey Shelton Okot,

As you said i had changed the setting in /etc/hosts, but you had mentioned 192.166.66.66 as your local hosts, what do i do in my case. Highlighted is my failed region server node kindly suggest the ip which need to be added with this.

PFA

Regards,

Mathivanan

42398-hbase-hosts-change.png

42397-hbase-hosts.png

Re: Hbase Regions Server getting down due to connection refused error

@Mathi Murugan,

I would suggest you to try changing hostnames to lower case in /etc/hosts in all the ambari nodes and restart ambari agents and ambari server.

Make sure that "hostname -f" gives lower case hostname and the same is used in /etc/hosts

Change your /etc/hosts entries as below

10.... biz-scul-hadoop-nn 
...

Also make sure that the below call gives hostnames in lower case

curl -k -u {username}:{password} http://{ambari-host}:{ambari-port}/api/v1/hosts

Thanks,

Aditya