Cannot connect to ResoureManager

[#1]

Since HiveUI is not working through Ambari, on troubleshooting through "hive -hiveconf hive.root.logger=DEBUG,console",

16/04/13 06:44:37 [IPC Client (1720891078) connection to RM_FQDN /RM_IP:8020 from root]: DEBUG ipc.Client: IPC Client (1720891078) connection to RM_FQDN/RM_IP:8020 from root got value #32

16/04/13 06:44:37 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms

16/04/13 06:44:37 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.

16/04/13 06:44:37 [main]: DEBUG ipc.Client: Connecting to RM_FQDN/RM_IP:8050

16/04/13 06:44:38 [main]: INFO ipc.Client: Retrying connect to server: RM_FQDN/RM_IP:8050. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)

16/04/13 06:44:39 [main]: INFO ipc.Client: Retrying connect to server: RM_FQDN/RM_IP:8050. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)

Also ResourceManager UI giving "ERR_CONNECTION_REFUSED"

Seems even when RM is already started through Ambari, it cannot be accessed/connected through client or UI.

[#2]

can you please check the RM logs for any possible errors?

[#3]

ResourceManager should not be on port 8020, default is 8088. 8020 is used by NameNode.

[#4]

@Amit Tewari

1. Can you please check if resource manager is listening on 8050

e.g.

[root@sandbox coord]# netstat -tulpn|grep 8050
tcp        0      0 0.0.0.0:8050                0.0.0.0:*                   LISTEN      8102/jav

2. If yes, can you please check if is reachable from hive client ? you can try simple ping command first and if ping is working then try to do telnet from hive client

e.g.

[root@sandbox coord]# telnet sandbox.hortonworks.com 8050
Trying 10.0.2.15...
Connected to sandbox.hortonworks.com.
Escape character is '^]'.

3. If you still get connection refused then try telnet from some other node in the cluster, if it works then your hive client has some connectivity issues with RM

4. If it doesn't work from any of the host then RM has some issues, you might need to check RM logs or RM host if any firewall is blocking packets to 8050.

Hope this helps.

[#5]

@Kuldeep Kulkarni Couldnt connect using telnet

[#6]

@Amit Tewari

Is netstat working? please paste output of netstat command here

[#7]

@Amit Tewari - Please check my answer below and accept if this resolves your issue

[#8 · Accepted solution]

Here's how I resolved the issue:

RM log was showing some issue related to ZK

:2016-04-18 05:20:57,275 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1214)) - Retrying operation on ZK. Retry no. 195 2016-04-18 05:20:57,454 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server ZK_HOST/10.131.$ 2016-04-18 05:20:57,455 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1146)) - Session 0x0 for server null, unexpected error, closing socket connection and attemptin$ java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)

Also noticed ZK kept getting stopped. ZK logs: java.io.FileNotFoundException: /hadoop/zookeeper/version-2/log. (Permission denied)

So did chmod a+rwx -R /hadoop/zookeeper andd then restarted ZK and RM services.

After that was able to successfully start Hive CLI.

Thanks @Kuldeep Kulkarni @Laurent Edel