Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HiveServer can't start after switch on Kerberos

Solved Go to solution
Highlighted

HiveServer can't start after switch on Kerberos

Try to switch on Kerberos with manual creation of keytabs and principals.

Principal and keytabs has been created. Keytabs distributed to all nodes. Almost all services started.

Hive, HBase and Kafk don't.

Now I try to solve Hive issue.

hiveserver2.log

2016-10-24 16:57:10,382 ERROR [Thread-15-SendThread(DN2.REALM.COM:2181)]: client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:createSaslToken(384)) - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2016-10-24 16:57:10,382 ERROR [Thread-15-SendThread(DN2.REALM.COM:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:run(1059)) - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2016-10-24 16:57:10,383 ERROR [Thread-15-EventThread]: curator.ConnectionState (ConnectionState.java:checkState(245)) - Authentication failed
2016-10-24 16:57:10,390 FATAL [Thread-15]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegation/HIVESERVER2/keys
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
        at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:128)
        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hive/cluster/delegation/HIVESERVER2/keys
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
        ... 6 more
2016-10-24 16:57:10,399 INFO  [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(711)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at DN1.REALM.COM/172.26.140.200
************************************************************/
2016-10-24 16:57:10,419 INFO  [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(249)) - Created a znode on ZooKeeper for HiveServer2 uri: DN1.REALM.COM:10000
2016-10-24 16:57:10,420 INFO  [Thread-13]: server.HiveServer2 (HiveServer2.java:stop(397)) - Shutting down HiveServer2
2016-10-24 16:57:10,420 INFO  [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-10-24 16:57:10,420 INFO  [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-10-24 16:57:10,420 INFO  [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: HiveServer can't start after switch on Kerberos

Super Guru

@Nikita Kiselev

Can you please double check your principal name?

By looking at below message, it looks like we have wrong hostname part in the principal or some typo etc.

Mechanism level:Servernot found inKerberos database (7))

Also,

Try to do kinit using hive service keytab and see if it works?

View solution in original post

2 REPLIES 2
Highlighted

Re: HiveServer can't start after switch on Kerberos

Super Guru

@Nikita Kiselev

Can you please double check your principal name?

By looking at below message, it looks like we have wrong hostname part in the principal or some typo etc.

Mechanism level:Servernot found inKerberos database (7))

Also,

Try to do kinit using hive service keytab and see if it works?

View solution in original post

Highlighted

Re: HiveServer can't start after switch on Kerberos

kinit is working well.

Problem solved adter reading this

https://hortonworks.jira.com/browse/BUG-42602

and change hive-site.xml

hive.cluster.delegation.token.store.class=org.apache.hadoop.hive.thrift.DBTokenStore
Don't have an account?
Coming from Hortonworks? Activate your account here