Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kerberized Hiverserver2 shutting down

Highlighted

Kerberized Hiverserver2 shutting down

Contributor

HDP 2.4 / Amabri 2.2.1.0

-------------------------------------------

My hiveserver2 is shutting down after restart. This is happening after implementing MIT KDC Kerberos.

2016-05-07 16:59:34,513 FATAL [Thread-13]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationHIVESERVER2/keys at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge.java:444) at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:124) at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationHIVESERVER2/keys at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:672) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423) at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257) at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160) ... 6 more 2016-05-07 16:59:34,516 INFO [Thread-9]: provider.AuditProviderFactory (AuditProviderFactory.java:run(454)) - ==> JVMShutdownHook.run() 2016-05-07 16:59:34,517 INFO [Thread-11]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2

4 REPLIES 4

Re: Kerberized Hiverserver2 shutting down

hive.cluster.delegation.token.store.zookeeper.znode is the root path for token store data and by default set to /hive/cluster/delegation. However, Hive tries to create path "/hive/cluster/delegationHIVESERVER2/keys", it looks like a slash is missing, can you try to set

hive.cluster.delegation.token.store.zookeeper.znode=/hive/cluster/delegation/

with a slash appended at the end. Can you also check your hive.cluster.delegation.token.store.zookeeper.connectString, it should point to your ZK quorum, like "server1:2181,server2:2181,server3:2181".

Re: Kerberized Hiverserver2 shutting down

Contributor

@Predrag Minovic

Thanks for the support. I made the changes. Now still the error is same:

Error creating path /hive/cluster/delegation/HIVESERVER2/keys

Re: Kerberized Hiverserver2 shutting down

Expert Contributor

Can you also check value for hive.zookeeper.quorum? roughly scan through hive config to see if any key value is changed to localhost instead of valid hostname. You can do a Hive previous and current config compare through Ambari.

Regards

Pranay Vuas

Re: Kerberized Hiverserver2 shutting down

Contributor

@Pranay Vyas

Yes, I checked the ZK quorum. There is no localhost. I have 5 ZK with proper FQDN:2181.