@Ramon Wartala
Hi Ramon,
I faced the same issue after one week of cluster being down on a HDP 2.6.2. When trying to start it again, all services refuse to start except Zookeeper and the smoke test fails like yours.
I am using a KDC and the zookeeper principals are :
kadmin.local -q "listprincs" | grep zookeeper
zookeeper/ip-10-2-2-136.eu-central-1.compute.internal@HDPBASE
zookeeper/ip-10-2-2-181.eu-central-1.compute.internal@HDPBASE
zookeeper/ip-10-2-2-43.eu-central-1.compute.internal@HDPBASE
The smoke test fails with the following errors:
2017-11-24 09:28:07,807 - File['/var/lib/ambari-agent/tmp/zkSmoke.out'] {'action': ['delete']}
2017-11-24 09:28:07,807 - File['/var/lib/ambari-agent/tmp/zkSmoke.sh'] {'content': StaticFile('zkSmoke.sh'), 'mode': 0755}
2017-11-24 09:28:07,808 - Execute['/var/lib/ambari-agent/tmp/zkSmoke.sh /usr/hdp/current/zookeeper-client/bin/zkCli.sh ambari-qa /usr/hdp/current/zookeeper-client/conf 2181 True /usr/bin/kinit /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-hdpbase@HDPBASE /var/lib/ambari-agent/tmp/zkSmoke.out'] {'logoutput': True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'try_sleep': 5}
zk_node1=ip-10-2-2-136.eu-central-1.compute.internal
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /zk_smoketest
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:703)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:363)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
I have checked the hostname on all nodes and the hostname returns the short name, while hostname -f returns the fqdn.
I changed the ambari-agent.ini on all node to use a hostname_script that returns hostname -f since I previously had an issue with the ambari heartbeats.
Could you resolve the issue ?
I already tried regenerating keytabs and disabling/reenabling Kerberos without any success.
Best Regards,
Eric Le Blouc'h
