Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

RM ui

Hi Friends ,

RM UI page is not accessible. log is below.

[root@node1 yarn]# netstat -anp | grep 8088
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 111749/java
tcp 0 0 10.160.102.162:56606 10.160.102.162:8088 TIME_WAIT -
tcp 0 0 10.160.102.162:56656 10.160.102.162:8088 TIME_WAIT -
tcp 0 0 10.160.102.162:56604 10.160.102.162:8088 TIME_WAIT -
tcp 0 0 10.160.102.162:56610 10.160.102.162:8088 TIME_WAIT -
tcp 0 0 10.160.102.162:56618 10.160.102.162:8088 TIME_WAIT -

=========================================================================

 

2021-03-12 17:54:39,512 INFO client.RpcRetryingCallerImpl (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=6, retries=6, started=4117 ms ago, cancelled=false, msg=Call to host35.fifo.com/10.211.111.17:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifo.com/10.211.111.17:17020, details=row 'prod.timelineservice.flowrun,yarn-cluster!svcumf1d!distcp!^?�����ӕ,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host35.fifo.com,17020,1590141203044, seqNum=-1
2021-03-12 17:54:43,525 INFO client.RpcRetryingCallerImpl (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=7, retries=7, started=8130 ms ago, cancelled=false, msg=Call to host35.fifo.com/10.211.111.17:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifo.com/10.211.111.17:17020, details=row 'prod.timelineservice.flowrun,yarn-cluster!svcumf1d!distcp!^?�����ӕ,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host35.fifo.com,17020,1590141203044, seqNum=-1
2021-03-12 17:54:46,665 INFO webapp.ApiServer (ApiServer.java:getVersion(113)) - 3.1.1.3.1.0.0-78 from e4f82af51faec922b4804d0232a637422ec29e64 by jenkins source checksum eab9fa2a6aa38c6362c66d8df75774
2021-03-12 17:54:46,759 INFO webapp.ApiServer (ApiServer.java:getService(197)) - GET: getService for appName = ats-hbase user = yarn-ats (auth:PROXY) via rm/host15.fifo.com@CORPDEV.fifo.com (auth:KERBEROS)
2021-03-12 17:54:46,764 INFO client.RequestHedgingRMFailoverProxyProvider (RequestHedgingRMFailoverProxyProvider.java:init(89)) - Created wrapped proxy for [rm1, rm2]
2021-03-12 17:54:46,765 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at host15.fifo.com/10.160.102.162:10200
2021-03-12 17:54:46,765 INFO utils.ServiceApiUtil (ServiceApiUtil.java:loadService(359)) - Loading service definition from hdfs://UMFDEV/user/yarn-ats/.yarn/services/ats-hbase/ats-hbase.json
2021-03-12 17:54:46,819 INFO client.RequestHedgingRMFailoverProxyProvider (RequestHedgingRMFailoverProxyProvider.java:invoke(147)) - Looking for the active RM in [rm1, rm2]...
2021-03-12 17:54:46,851 INFO ipc.Server (Server.java:logException(2719)) - IPC Server handler 11 on 8032, call Call#32992 Retry#0 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport from 10.160.102.162:39836: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1590152319040_0001' doesn't exist in RM. Please check that the job submission was successful.
2021-03-12 17:54:46,857 WARN client.RequestHedgingRMFailoverProxyProvider (RequestHedgingRMFailoverProxyProvider.java:invoke(177)) - Invocation returned exception: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1590152319040_0001' doesn't exist in RM. Please check that the job submission was successful.
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:382)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:234)
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:561)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
on [rm1], so propagating back to caller.
2021-03-12 17:54:46,879 ERROR webapp.ApiServer (ApiServer.java:getService(216)) - Get service failed: {}
java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1748)
at org.apache.hadoop.yarn.service.webapp.ApiServer.getServiceFromClient(ApiServer.java:745)
"hadoop-yarn-resourcemanager-host15.fifo.com.log" 936672L, 184034377C

 

Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifa.com/10.211.111.17:17020
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
... 7 more

 

Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifa.com/10.211.111.17:17020
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
... 7 more
Caused by: java.net.ConnectException: Connection refused
... 11 more
2021-03-18 20:11:17,107 INFO client.RpcRetryingCallerImpl (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=6, retries=6, started=4132 ms ago, cancelled=false, msg=Call to host35.fifa.com/10.211.111.17:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifa.com/10.211.111.17:17020, details=row 'prod.timelineservice.flowrun,yarn-cluster!svcumf1d!distcp!^?����Xb�,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host35.fifa.com,17020,1590141203044, seqNum=-1

3 REPLIES 3

@mohammad_shamim  Have you enabled Kerberos in the cluster? Can you paste the screenshot of RM UI? Is there any error do you see on the page?

Yes Kerberos is enabled. 

Do you have SPNEGO enabled for browsers? 

 

https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/authentication-with-kerberos/content/authe_spn...

 

Are you seeing any error on the UI?