After HDP 2.4.2 to 2.5.3 upgrade, standby resource manager is checked first

[#1]

[root@jtldjob ~]# yarn application -list 17/02/04 10:33:34 INFO impl.TimelineClientImpl: Timeline service address: http://str20:8188/ws/v1/timeline/ 17/02/04 10:33:34 INFO client.AHSProxy: Connecting to Application History server at str20/10.5.168.121:10200 17/02/04 10:33:35 WARN ipc.Client: Failed to connect to server: str20:8032: retries get failed due to exceeded maximum allowed retries number: 0 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:650) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:745) at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618) at org.apache.hadoop.ipc.Client.call(Client.java:1449) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy17.getApplications(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:251) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy18.getApplications(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:484) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.listApplications(ApplicationCLI.java:401) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:207) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:83) 17/02/04 10:33:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):1 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1485795502013_1714 HIVE-8dc9a187-2c8c-44b3-92a9-eec0662e524b TEZ talend ServAssure RUNNING UNDEFINED 77.83% http://str44/ui/

[#2]

You can safely ignore this. Your app first tried to contact rm1 and found it to be in the Stand-By mode: "WARN ipc.Client: Failed to connect to server: str20/10.5.168.121:8032:". After that it failed over to rm2: "INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2". If rm1 is Active, there will be no such message.

[#3]

@Predrag Minovic

I understand what you are saying but how can I change this to contact the active RM first? And how come this worked in 2.4.2 and not in 2.5.3, there should be some parameter changes?? Also, everytime it contacts resource manager, it is wasting some time checking which is active..

[#4 · Accepted solution]

@PJ

Regarding your question - "I understand what you are saying but how can I change this to contact the active RM first? And how come this worked in 2.4.2 and not in 2.5.3, there should be some parameter changes?? Also, everytime it contacts resource manager, it is wasting some time checking which is active.."

You can make "rm1" as active using failover command.

Please use below command to failover from rm2 to rm1 as 'yarn' user on any yarn-client

yarn rmadmin -failover rm2 rm1

If you have enabled automatic failover and for some reason above command fails then use below command( If you are doing it for production, please be very careful or contact Hortonworks Support )

yarn rmadmin -transitionToActive rm1 --forceactive --forcemanual

OR

If no jobs are running, then simply restart "rm2" from Ambari, rm1 will automatically become Active if automatic failover is enabled.