HiveServer can't start after switch on Kerberos

nord_tramper — Mon, 24 Oct 2016 21:20:25 GMT

Try to switch on Kerberos with manual creation of keytabs and principals.

Principal and keytabs has been created. Keytabs distributed to all nodes. Almost all services started.

Hive, HBase and Kafk don't.

Now I try to solve Hive issue.

hiveserver2.log

2016-10-24 16:57:10,382 ERROR [Thread-15-SendThread(DN2.REALM.COM:2181)]: client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:createSaslToken(384)) - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2016-10-24 16:57:10,382 ERROR [Thread-15-SendThread(DN2.REALM.COM:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:run(1059)) - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2016-10-24 16:57:10,383 ERROR [Thread-15-EventThread]: curator.ConnectionState (ConnectionState.java:checkState(245)) - Authentication failed
2016-10-24 16:57:10,390 FATAL [Thread-15]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegation/HIVESERVER2/keys
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
        at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:128)
        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hive/cluster/delegation/HIVESERVER2/keys
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
        ... 6 more
2016-10-24 16:57:10,399 INFO  [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(711)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at DN1.REALM.COM/172.26.140.200
************************************************************/
2016-10-24 16:57:10,419 INFO  [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(249)) - Created a znode on ZooKeeper for HiveServer2 uri: DN1.REALM.COM:10000
2016-10-24 16:57:10,420 INFO  [Thread-13]: server.HiveServer2 (HiveServer2.java:stop(397)) - Shutting down HiveServer2
2016-10-24 16:57:10,420 INFO  [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-10-24 16:57:10,420 INFO  [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-10-24 16:57:10,420 INFO  [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.

Re: HiveServer can't start after switch on Kerberos

KuldeepK — Mon, 24 Oct 2016 21:57:06 GMT

@Nikita Kiselev

Can you please double check your principal name?

By looking at below message, it looks like we have wrong hostname part in the principal or some typo etc.

Mechanism level:Servernot found inKerberos database (7))

Also,

Try to do kinit using hive service keytab and see if it works?

Re: HiveServer can't start after switch on Kerberos

nord_tramper — Mon, 24 Oct 2016 22:05:41 GMT

kinit is working well.

Problem solved adter reading this

https://hortonworks.jira.com/browse/BUG-42602

and change hive-site.xml

hive.cluster.delegation.token.store.class=org.apache.hadoop.hive.thrift.DBTokenStore

question HiveServer can't start after switch on Kerberos in Archives of Support Questions (Read Only)

HiveServer can't start after switch on Kerberos

Re: HiveServer can't start after switch on Kerberos

Re: HiveServer can't start after switch on Kerberos