rolling upgrade HDP 2.2.4 to 2.3.4 issue on Zookeeper component

[#1]

Hi,

after covering prereq. for a rolling upgrade of HDP 2.2.4 to 2.3.4 (using Ambari 2.2.1.1) and starting the procedure, the first component, Zookeeper, fails with below error ("##zookeeperserver1##" is just a placeholder for real server name):

...

2016-06-03 13:42:07,725 - Will retry 1 time(s), caught exception: Failed to establish zookeeper quorum. Sleeping for 2 sec(s)
2016-06-03 13:42:09,727 - call['echo 'create /id1facad4c_date410316 mydata' | /usr/hdp/current/zookeeper-server/bin/zkCli.sh -server ##zookeeperserver1##:2181'] {'logoutput': True, 'user': 'zookeeper', 'quiet': False}
Connecting to ##zookeeperserver1##:2181
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:AuthFailed type:None path:null
[zk: ##zookeeperserver1##:2181(CONNECTING) 0] create /id1facad4c_date410316 mydata

WATCHER::

WatchedEvent state:Disconnected type:None path:null
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /id1facad4c_date410316
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:698)
	at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
	at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:363)
	at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
	at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
2016-06-03 13:42:10,174 - call returned (1, 'Connecting to ##zookeeperserver1##:2181\nWelcome to ZooKeeper!\nJLine support is enabled\n\nWATCHER::\n\nWatchedEvent state:AuthFailed type:None path:null\n[zk: ##zookeeperserver1##:2181(CONNECTING) 0] create /id1facad4c_date410316 mydata\n\nWATCHER::\n\nWatchedEvent state:Disconnected type:None path:null\nException in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /id1facad4c_date410316\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:99)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:51)\n\tat org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)\n\tat org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:698)\n\tat org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)\n\tat org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:363)\n\tat org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)\n\tat org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)')

What to do to be able to proceed with the upgrade ?!?!

Thanks, Gerd

[#2]

@Gerd Koenig

It seems like the authentication was failed with zookeeper. Through which user you are perform the upgrade and is this a secure cluster?

WatchedEvent state:AuthFailed type:None path:null

[#3 · Accepted solution]

Hi @Jitendra Yadav , thanks for getting back to me.

In the meantime the issue got solved recently through a support call.....

It looks like the upgrade process ran into a timeout while talking to Zookeeper. To proceed with the upgrade, the solution was to do "Pause Upgrade", go to Ambari Dashboard, check state of Zookeeper (which was fine, all up and running), then click "Resume Upgrade". After that the problem was gone and upgrade procedure continued.

Best regards, Gerd