I have faced with similar error as here: https://community.hortonworks.com/questions/28589/hive-metastore-wont-start-after-enabling-kerberos.... (due to message size limitations, couldn't comment there)
Cluster layout:
NodeA - majority of hadoop services
NodeB - hadoop clients installed. Kerberos installed.
Stack:
OS:
Kerberos 5 version 1.15.1
Ambari 2.4.1.0
HDP: 2.5.0.0
-- hive 1.2.1.2.5
-- zookeeper 3.4.6.2.5
-- kerberos 1.10.3-10
How to reproduce:
1. Unkerberized cluster is deployed it works fine.
2. I kerberize the cluster: all services are up, except hive metastore (shown as start is successful, but fails immediately after start)
Additional info:
a. If I unkeberize the cluster - it works fine again.
b. In zookeeper there is not created even /hive znode. The links above were checked and when I have added property "hive.cluster.delegation.token.store.zookeeper.acl=sasl:hive:cdrwa"
then (after the change) I restarted the hive services: /hive znode was created with named properties, but it was empty. The ACL were setup as above. The /hive znode was accessible using /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM
c. The configs are mostly left as default. The most relevant to the issue are here:
hive.metastore.kerberos.keytab.file = /etc/security/keytabs/hive.service.keytab
hive.metastore.kerberos.principal = hive/_HOST@SOMEREALM
hive.metastore.sasl.enabled = true
hive.server2.authentication.kerberos.principal = hive/_HOST@SOMEREALM
hive.server2.authentication.spnego.keytab = /etc/security/keytabs/spnego.service.keytab
hive.server2.authentication.spnego.principal = HTTP/_HOST@SOMEREALM
templeton.hive.properties = hive.metastore.local=false,hive.metastore.uris=thrift://<some-address>:9083,hive.metastore.sasl.enabled=true,hive.metastore.execute.setugi=true,hive.metastore.warehouse.dir=/apps/hive/warehouse,hive.exec.mode.local.auto=false,hive.metastore.kerberos.principal=hive/_HOST@SOMEREALM
atlas.jaas.KafkaClient.option.principal = hive/_HOST@SOMEREALM
hive.llap.zk.sm.principal = hive/_HOST@SOMEREALM
hive.llap.daemon.service.principal = hive/_HOST@SOMEREALM
xasecure.audit.jaas.Client.option.principal = hive/_HOST@SOMEREALM
templeton.kerberos.principal = HTTP/_HOST@SOMEREALM
hive.cluster.delegation.token.store.class = org.apache.hadoop.hive.thrift.ZooKeeperTokenStore
d. kinit was tried (I expect to try them all, but let me know if some to be double checked):
zookeeper user:
kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.service.keytab hive/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM
hive user:
kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.service.keytab hive/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM
e. The error in hivemetastore logs:
2018-01-25 12:30:45,342 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6326)) - org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6241)
at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys
at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
... 11 more
2018-01-25 12:30:45,343 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(6159)) - Metastore Thrift Server threw an exception...
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6241)
at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys
at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
... 11 more
2018-01-25 12:30:45,395 INFO [Thread-4]: metastore.HiveMetaStore (HiveMetaStore.java:run(6125)) - Shutting down hive metastore.
