Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ZKFailoverController not starting after Kerbores Disabled

Highlighted

ZKFailoverController not starting after Kerbores Disabled

Explorer

Hi Team,

After Kerbores disable iam not able to start the Zookeeper failover controler service. Getting below error. while trying to start

_________________

2016-10-17 06:14:08,456 WARN client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:handle(492)) - Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock.

2016-10-17 06:14:08,460 WARN zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1001)) - SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.

2016-10-17 06:14:08,464 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server rdlc000019.rlab.discoverfinancial.com/10.150.32.93:2181

2016-10-17 06:14:08,469 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to rdlc000019.rlab.discoverfinancial.com/10.150.32.93:2181, initiating session

2016-10-17 06:14:08,476 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1279)) - Session establishment complete on server rdlc000019.rlab.discoverfinancial.com/10.150.32.93:2181, sessionid = 0x157d1aed2d4000d, negotiated timeout = 5000

2016-10-17 06:14:08,483 FATAL ha.ActiveStandbyElector (ActiveStandbyElector.java:fatalError(690)) - Unexpected Zookeeper watch event state: AuthFailed

2016-10-17 06:14:08,483 INFO ha.ActiveStandbyElector (ActiveStandbyElector.java:terminateConnection(830)) - Terminating ZK connection for elector id=187283593 appData=null cb=Elector callbacks for NameNode at rdlc000020.rlab.discoverfinancial.com/10.150.32.94:8020

2016-10-17 06:14:08,490 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x157d1aed2d4000d closed

2016-10-17 06:14:08,491 FATAL ha.ZKFailoverController (ZKFailoverController.java:fatalError(374)) - Fatal error occurred:Unexpected Zookeeper watch event state: AuthFailed

2016-10-17 06:14:08,491 WARN ha.ActiveStandbyElector (ActiveStandbyElector.java:isStaleClient(1066)) - Ignoring stale result from old client with sessionId 0x157d1aed2d4000d

2016-10-17 06:14:08,491 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down

2016-10-17 06:14:08,505 INFO ipc.CallQueueManager (CallQueueManager.java:<init>(57)) - Using callQueue class java.util.concurrent.LinkedBlockingQueue

2016-10-17 06:14:08,531 INFO ipc.Server (Server.java:run(715)) - Starting Socket Reader #1 for port 8019

2016-10-17 06:14:08,567 INFO ipc.Server (Server.java:run(949)) - IPC Server Responder: starting

2016-10-17 06:14:08,570 INFO ipc.Server (Server.java:run(785)) - IPC Server listener on 8019: starting

2016-10-17 06:14:08,573 INFO ipc.Server (Server.java:stop(2617)) - Stopping server on 8019

2016-10-17 06:14:08,574 INFO ha.ActiveStandbyElector (ActiveStandbyElector.java:quitElection(401)) - Yielding from election

2016-10-17 06:14:08,574 INFO ha.ActiveStandbyElector (ActiveStandbyElector.java:terminateConnection(827)) - terminateConnection, zkConnectionState = TERMINATED

2016-10-17 06:14:08,574 INFO ha.HealthMonitor (HealthMonitor.java:shutdown(151)) - Stopping HealthMonitor thread

2016-10-17 06:14:08,574 INFO ipc.Server (Server.java:run(954)) - Stopping IPC Server Responder

2016-10-17 06:14:08,575 INFO ipc.Server (Server.java:run(817)) - Stopping IPC Server listener on 8019

2016-10-17 06:14:08,606 INFO ha.ActiveStandbyElector (ActiveStandbyElector.java:terminateConnection(827)) - terminateConnection, zkConnectionState = TERMINATED

2016-10-17 06:14:08,606 FATAL tools.DFSZKFailoverController (DFSZKFailoverController.java:main(185)) - Got a fatal error, exiting now

java.lang.RuntimeException: ZK Failover Controller failed: Unexpected Zookeeper watch event state: AuthFailed

at org.apache.hadoop.ha.ZKFailoverController.mainLoop(ZKFailoverController.java:369)

at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:238)

at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:61)

at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:172)

at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:168)

at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:449)

at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:168)

at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:183)

2016-10-17 06:14:08,609 INFO tools.DFSZKFailoverController (LogAdapter.java:info(45)) - SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down DFSZKFailoverController at rdlc000020.rlab.discoverfinancial.com/10.150.32.94

************************************************************

4 REPLIES 4
Highlighted

Re: ZKFailoverController not starting after Kerbores Disabled

Explorer

Team any answer

Highlighted

Re: ZKFailoverController not starting after Kerbores Disabled

Mentor

@suresh krish

How to delete a HA Zookeeper or this

Highlighted

Re: ZKFailoverController not starting after Kerbores Disabled

Super Guru
Highlighted

Re: ZKFailoverController not starting after Kerbores Disabled

Explorer

Even after Shutdown all zookeepers then restart one at a time, the zkfc start failed in HA mode, any other suggestion?

Don't have an account?
Coming from Hortonworks? Activate your account here