Hive Metastore won't start after enabling Kerberos. Znodes are not created

sergejs_andreje — Fri, 26 Jan 2018 19:43:29 GMT

I have faced with similar error as here: https://community.hortonworks.com/questions/28589/hive-metastore-wont-start-after-enabling-kerberos.html (due to message size limitations, couldn't comment there)

Cluster layout:
NodeA - majority of hadoop services

NodeB - hadoop clients installed. Kerberos installed.

Stack:

OS:

Kerberos 5 version 1.15.1

Ambari 2.4.1.0

HDP: 2.5.0.0

-- hive 1.2.1.2.5

-- zookeeper 3.4.6.2.5

-- kerberos 1.10.3-10

How to reproduce:

1. Unkerberized cluster is deployed it works fine.

2. I kerberize the cluster: all services are up, except hive metastore (shown as start is successful, but fails immediately after start)

Additional info:

a. If I unkeberize the cluster - it works fine again.

b. In zookeeper there is not created even /hive znode. The links above were checked and when I have added property "hive.cluster.delegation.token.store.zookeeper.acl=sasl:hive:cdrwa"

then (after the change) I restarted the hive services: /hive znode was created with named properties, but it was empty. The ACL were setup as above. The /hive znode was accessible using /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM

c. The configs are mostly left as default. The most relevant to the issue are here:

hive.metastore.kerberos.keytab.file = /etc/security/keytabs/hive.service.keytab
hive.metastore.kerberos.principal = hive/_HOST@SOMEREALM
hive.metastore.sasl.enabled = true
hive.server2.authentication.kerberos.principal = hive/_HOST@SOMEREALM
hive.server2.authentication.spnego.keytab = /etc/security/keytabs/spnego.service.keytab
hive.server2.authentication.spnego.principal = HTTP/_HOST@SOMEREALM
   
templeton.hive.properties = hive.metastore.local=false,hive.metastore.uris=thrift://<some-address>:9083,hive.metastore.sasl.enabled=true,hive.metastore.execute.setugi=true,hive.metastore.warehouse.dir=/apps/hive/warehouse,hive.exec.mode.local.auto=false,hive.metastore.kerberos.principal=hive/_HOST@SOMEREALM
atlas.jaas.KafkaClient.option.principal = hive/_HOST@SOMEREALM
   
hive.llap.zk.sm.principal = hive/_HOST@SOMEREALM
hive.llap.daemon.service.principal = hive/_HOST@SOMEREALM
   
xasecure.audit.jaas.Client.option.principal = hive/_HOST@SOMEREALM
templeton.kerberos.principal = HTTP/_HOST@SOMEREALM
   
hive.cluster.delegation.token.store.class = org.apache.hadoop.hive.thrift.ZooKeeperTokenStore

d. kinit was tried (I expect to try them all, but let me know if some to be double checked):

zookeeper user:
kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.service.keytab hive/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM

hive user:
kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.service.keytab hive/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM

e. The error in hivemetastore logs:

2018-01-25 12:30:45,342 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6326)) - org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6241)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
        ... 11 more

2018-01-25 12:30:45,343 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(6159)) - Metastore Thrift Server threw an exception...
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6241)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
        ... 11 more 

2018-01-25 12:30:45,395 INFO [Thread-4]: metastore.HiveMetaStore (HiveMetaStore.java:run(6125)) - Shutting down hive metastore.

Re: Hive Metastore won't start after enabling Kerberos. Znodes are not created

jsensharma — Sun, 18 Aug 2019 04:53:54 GMT

@Sergejs Andrejevs

Can you please check if your "hive-site.xml" has the following property defined and set to "true". I see it in your "templeton.hive.properties" but please verify your hive-site.xml as well.

hive.metastore.sasl.enabled =  true

Ambari UI --> Hive --> Configs --> Advanced --> Advanced hive-site

And then in the above location please check if the property "hive.metastore.sasl.enabled" is set or not? If not then please try setting it and followed by Hive Service restart.

Re: Hive Metastore won't start after enabling Kerberos. Znodes are not created

sergejs_andreje — Fri, 26 Jan 2018 20:07:55 GMT

hive.metastore.sasl.enabled = true

I'll update the initial post too with this info. Thanks for noting.

Re: Hive Metastore won't start after enabling Kerberos. Znodes are not created

sergejs_andreje — Tue, 30 Jan 2018 17:55:19 GMT

The issue looks to be within check of kerberos tickets: HiveMetastore wasn't using them.

Installed HDP2.5.3.0+ with the same configs and it worked.

question Re: Hive Metastore won't start after enabling Kerberos. Znodes are not created in Archives of Support Questions (Read Only)

Hive Metastore won't start after enabling Kerberos. Znodes are not created

Re: Hive Metastore won't start after enabling Kerberos. Znodes are not created

Re: Hive Metastore won't start after enabling Kerberos. Znodes are not created

Re: Hive Metastore won't start after enabling Kerberos. Znodes are not created