Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to start zookeeper failover controller in HA cluster and tried manually format the ZKFC on console using command(hdfs zkfc -formatZK)

Unable to start zookeeper failover controller in HA cluster and tried manually format the ZKFC on console using command(hdfs zkfc -formatZK)

Explorer

when manually tried to format the ZKFC on console using command(hdfs zkfc -formatZK)
getting failed with Couldn't create /hadoop-ha/dev

Any one have an idea about the below exception in ambari-metrics-collector.log
your input is much appreciated

please find the logs for reference

sudo hdfs zkfc -formatZK
20/03/24 16:08:49 INFO tools.DFSZKFailoverController: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DFSZKFailoverController
STARTUP_MSG: host = xxxx-0xx0xx0xx.xxxx.com/10.xx.xx.xx
STARTUP_MSG: args = [-formatZK]
STARTUP_MSG: version = 3.1.1.3.1.0.0-78
STARTUP_MSG: classpath =
20/03/24 16:08:49 INFO tools.DFSZKFailoverController: registered UNIX signal handlers for [TERM, HUP, INT]
20/03/24 16:08:49 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at xxxx-0xx0xx0xx.xxxx.com/10.xx.xx.xx:xx20
20/03/24 16:08:49 INFO security.UserGroupInformation: Login successful for user nn/xxxx-0xx0xx0xx.xxxx.com@XXXX.COM using keytab file /etc/security/keytabs/nn.service.keytab
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.1.2.el7.x86_64
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Client environment:user.name=root
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/xxxx.com/xxxxxxx
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=xxxx-0xx0xx0xx.xxxx.com:2181,xxxx-0xx0xx084.xxxx.com:2181,xxxx-0xx0xx094.xxxx.com:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@41d477ed
20/03/24 16:08:49 INFO zookeeper.Login: successfully logged in.
20/03/24 16:08:49 INFO zookeeper.Login: TGT refresh thread started.
20/03/24 16:08:49 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
20/03/24 16:08:49 INFO zookeeper.ClientCnxn: Opening socket connection to server xxxx-0xx0xx084.xxxx.com/10.xx.xx.84:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
20/03/24 16:08:49 INFO zookeeper.Login: TGT valid starting at: Tue Mar 24 16:08:49 UTC 2020
20/03/24 16:08:49 INFO zookeeper.Login: TGT expires: Wed Mar 25 02:08:49 UTC 2020
20/03/24 16:08:49 INFO zookeeper.Login: TGT refresh sleeping until: Wed Mar 25 00:09:10 UTC 2020
20/03/24 16:08:49 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.xx.xx.xx:38572, server: xxxx-0xx0xx084.xxxx.com/10.xx.xx.84:2181
20/03/24 16:08:49 INFO zookeeper.ClientCnxn: Session establishment complete on server xxxx-0xx0xx084.xxxx.com/10.xx.xx.84:2181, sessionid = 0x1710763e04xx1e6, negotiated timeout = 10000
20/03/24 16:08:49 INFO ha.ActiveStandbyElector: Session connected.
20/03/24 16:08:49 INFO ha.ActiveStandbyElector: Successfully authenticated to ZooKeeper using SASL.
20/03/24 16:08:49 ERROR ha.ZKFailoverController: The failover controller encounters runtime error
java.io.IOException: Couldn't create /hadoop-ha/dev-xx
at org.apache.hadoop.ha.ActiveStandbyElector.ensureParentZNode(ActiveStandbyElector.java:xx1)
at org.apache.hadoop.ha.ZKFailoverController.formatZK(ZKFailoverController.java:286)
at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:216)
at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:xx)
at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:175)
at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:3xx)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:4xx)
at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:171)
at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:195)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hadoop-ha/dev-xx
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.ha.ActiveStandbyElector$3.run(ActiveStandbyElector.java:10xx)
at org.apache.hadoop.ha.ActiveStandbyElector$3.run(ActiveStandbyElector.java:1033)
at org.apache.hadoop.ha.ActiveStandbyElector.zkDoWithRetries(ActiveStandbyElector.java:1098)
at org.apache.hadoop.ha.ActiveStandbyElector.zkDoWithRetries(ActiveStandbyElector.java:10xx)
at org.apache.hadoop.ha.ActiveStandbyElector.createWithRetries(ActiveStandbyElector.java:1033)
at org.apache.hadoop.ha.ActiveStandbyElector.ensureParentZNode(ActiveStandbyElector.java:350)
... 11 more
20/03/24 16:08:49 INFO zookeeper.ZooKeeper: Session: 0x1710763e04xx1e6 closed
20/03/24 16:08:49 INFO zookeeper.ClientCnxn: EventThread shut down
20/03/24 16:08:49 FATAL tools.DFSZKFailoverController: DFSZKFailOverController exiting due to earlier exception java.io.IOException: Couldn't create /hadoop-ha/dev-xx
20/03/24 16:08:49 INFO util.ExitUtil: Exiting with status 1: java.io.IOException: Couldn't create /hadoop-ha/dev-xx
20/03/24 16:08:49 INFO tools.DFSZKFailoverController: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at xxxx-0xx0xx0xx.xxxx.com/10.xx.xx.xx
************************************************************/

Thanks in Advance

Don't have an account?
Coming from Hortonworks? Activate your account here