Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive Metastore won't start after enabling Kerberos

Hive Metastore won't start after enabling Kerberos

New Contributor

I'm setting up a test cluster to match a client site and so it's HDP 2.2.4 with Ambari 2.1.0. 3 masters and 5 slaves, Centos 6, primarily HDFS, Yarn, Hive, HBase, Oozie and ZK. HDFS is in HA. Cluster was working fine pre-kerberos.

After enabling kerberos (fresh MIT KDC) everything seems to be happy except Hive - the metastore won't come up. The hivemetastore.log says:

2016-04-20 17:10:10,976 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(5654)) - Metastore Thrift Server threw an exception...

org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys

at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:165)

at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:235)

at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:468)

at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge20S.java:438)

at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:5724)

at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5650)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys

at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)

at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)

at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)

at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)

at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)

at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)

at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)

at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)

at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)

at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)

at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:159)

... 11 more

I've checked zookeeper and the only node under hive is /hive/cluster/delegationHIVESERVER2

Any advice on where to start?

22 REPLIES 22

Re: Hive Metastore won't start after enabling Kerberos

Hi @Oliver Meyn,

What version of Java are you running?

Re: Hive Metastore won't start after enabling Kerberos

New Contributor

Should have said, sorry: oracle jdk 1.8.0_40 (this originally said openjdk but I was wrong)

Re: Hive Metastore won't start after enabling Kerberos

Can you log into one of your ZK nodes, kinit as the ZK principal so you can launch the zkCli, and check the ACLs for the znode in question? Curious if something is off with the ACLs that is preventing the node from being created.

Re: Hive Metastore won't start after enabling Kerberos

Contributor

Alternatively , try updating hive-site.xml with new delegation token.

Re: Hive Metastore won't start after enabling Kerberos

Super Guru
@Oliver MeynCan you try changing below property in hive-site.xml and restart HiveMetastore?
hive.cluster.delegation.token.store.zookeeper.znode

Re: Hive Metastore won't start after enabling Kerberos

New Contributor

I tried changing it to '/unsecure-hive/cluster/delegation' with acl like:

[zk: hwmaster1(CONNECTED) 8] getAcl /unsecure-hive/cluster/delegation

'world,'anyone

: cdrwa

But got the same error.

Re: Hive Metastore won't start after enabling Kerberos

Cloudera Employee

You can get it working by doing a manual kinit from the keytab using "kinit -k -t <file> <hive principal>"

There is a fix (HIVE-9685) that was put into HDP 2.2.5.x.

Re: Hive Metastore won't start after enabling Kerberos

New Contributor

Thanks @owen but I'm not sure where to kinit - ambari is doing the metastore restart so I don't know where to hook in. Also that JIRA appears to be CLI related so not sure if it's the right thing.

Highlighted

Re: Hive Metastore won't start after enabling Kerberos

Contributor

@Oliver Meyn

I am also facing the exact same issue. Did you solved this, if yes, please let me know what is the solution. Thanks