hive cli access taking long time if standby namenode is unavailable

[#1]

In the event when active namenode fails over to standby node which becomes active, if the standby node is not started , users accessing hive via hive cli take long time to connect or the connection never happens.

When checking debug log via following command :

hive --hiveconf hive.root.logger=DEBUG,console

Following message appears multiple times in the console output:

INFO ipc.Client: Retrying connect to server: rp1.HORTON/172.26.97.110:8020. Already tried 0 time(s); retry policy is RetryPolicy[MultipleLinearRandomRetry[500x2000ms], TryOnceThenFail]

DEBUG retry.RetryUtils: RETRY 1) policy=MultipleLinearRandomRetry, exception=java.net.ConnectException: Connection refused

Even after changing the property "yarn.resourcemanager.fs.state-store.retry-policy-spec" to lower value , MultipleLinearRandomRetry policy still remains the same and hive cli does not connect or connection never happens.

FIX:

Inorder to fix this issue , please update yarn-site.xml file with following property. This would require YARN service restart.

yarn.timeline-service.entity-group-fs-store.retry-policy-spec=1000, 1

After this change , try connecting via hive cli , the initial request will take some time however the subsequent requests would be faster.