Hi Friends ,
RM UI page is not accessible. log is below.
[root@node1 yarn]# netstat -anp | grep 8088
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 111749/java
tcp 0 0 10.160.102.162:56606 10.160.102.162:8088 TIME_WAIT -
tcp 0 0 10.160.102.162:56656 10.160.102.162:8088 TIME_WAIT -
tcp 0 0 10.160.102.162:56604 10.160.102.162:8088 TIME_WAIT -
tcp 0 0 10.160.102.162:56610 10.160.102.162:8088 TIME_WAIT -
tcp 0 0 10.160.102.162:56618 10.160.102.162:8088 TIME_WAIT -
=========================================================================
2021-03-12 17:54:39,512 INFO client.RpcRetryingCallerImpl (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=6, retries=6, started=4117 ms ago, cancelled=false, msg=Call to host35.fifo.com/10.211.111.17:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifo.com/10.211.111.17:17020, details=row 'prod.timelineservice.flowrun,yarn-cluster!svcumf1d!distcp!^?�����ӕ,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host35.fifo.com,17020,1590141203044, seqNum=-1
2021-03-12 17:54:43,525 INFO client.RpcRetryingCallerImpl (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=7, retries=7, started=8130 ms ago, cancelled=false, msg=Call to host35.fifo.com/10.211.111.17:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifo.com/10.211.111.17:17020, details=row 'prod.timelineservice.flowrun,yarn-cluster!svcumf1d!distcp!^?�����ӕ,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host35.fifo.com,17020,1590141203044, seqNum=-1
2021-03-12 17:54:46,665 INFO webapp.ApiServer (ApiServer.java:getVersion(113)) - 3.1.1.3.1.0.0-78 from e4f82af51faec922b4804d0232a637422ec29e64 by jenkins source checksum eab9fa2a6aa38c6362c66d8df75774
2021-03-12 17:54:46,759 INFO webapp.ApiServer (ApiServer.java:getService(197)) - GET: getService for appName = ats-hbase user = yarn-ats (auth:PROXY) via rm/host15.fifo.com@CORPDEV.fifo.com (auth:KERBEROS)
2021-03-12 17:54:46,764 INFO client.RequestHedgingRMFailoverProxyProvider (RequestHedgingRMFailoverProxyProvider.java:init(89)) - Created wrapped proxy for [rm1, rm2]
2021-03-12 17:54:46,765 INFO client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at host15.fifo.com/10.160.102.162:10200
2021-03-12 17:54:46,765 INFO utils.ServiceApiUtil (ServiceApiUtil.java:loadService(359)) - Loading service definition from hdfs://UMFDEV/user/yarn-ats/.yarn/services/ats-hbase/ats-hbase.json
2021-03-12 17:54:46,819 INFO client.RequestHedgingRMFailoverProxyProvider (RequestHedgingRMFailoverProxyProvider.java:invoke(147)) - Looking for the active RM in [rm1, rm2]...
2021-03-12 17:54:46,851 INFO ipc.Server (Server.java:logException(2719)) - IPC Server handler 11 on 8032, call Call#32992 Retry#0 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport from 10.160.102.162:39836: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1590152319040_0001' doesn't exist in RM. Please check that the job submission was successful.
2021-03-12 17:54:46,857 WARN client.RequestHedgingRMFailoverProxyProvider (RequestHedgingRMFailoverProxyProvider.java:invoke(177)) - Invocation returned exception: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1590152319040_0001' doesn't exist in RM. Please check that the job submission was successful.
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:382)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:234)
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:561)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
on [rm1], so propagating back to caller.
2021-03-12 17:54:46,879 ERROR webapp.ApiServer (ApiServer.java:getService(216)) - Get service failed: {}
java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1748)
at org.apache.hadoop.yarn.service.webapp.ApiServer.getServiceFromClient(ApiServer.java:745)
"hadoop-yarn-resourcemanager-host15.fifo.com.log" 936672L, 184034377C
Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifa.com/10.211.111.17:17020
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
... 7 more
Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifa.com/10.211.111.17:17020
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
... 7 more
Caused by: java.net.ConnectException: Connection refused
... 11 more
2021-03-18 20:11:17,107 INFO client.RpcRetryingCallerImpl (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=6, retries=6, started=4132 ms ago, cancelled=false, msg=Call to host35.fifa.com/10.211.111.17:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host35.fifa.com/10.211.111.17:17020, details=row 'prod.timelineservice.flowrun,yarn-cluster!svcumf1d!distcp!^?����Xb�,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host35.fifa.com,17020,1590141203044, seqNum=-1
