I have faced with similar error as here: https://community.hortonworks.com/questions/28589/hive-metastore-wont-start-after-enabling-kerberos.html (due to message size limitations, couldn't comment there) Cluster layout: NodeA - majority of hadoop services NodeB - hadoop clients installed. Kerberos installed. Stack: OS: Kerberos 5 version 1.15.1 Ambari 2.4.1.0 HDP: 2.5.0.0 -- hive 1.2.1.2.5 -- zookeeper 3.4.6.2.5 -- kerberos 1.10.3-10 How to reproduce: 1. Unkerberized cluster is deployed it works fine. 2. I kerberize the cluster: all services are up, except hive metastore (shown as start is successful, but fails immediately after start) Additional info: a. If I unkeberize the cluster - it works fine again. b. In zookeeper there is not created even /hive znode. The links above were checked and when I have added property "hive.cluster.delegation.token.store.zookeeper.acl=sasl:hive:cdrwa" then (after the change) I restarted the hive services: /hive znode was created with named properties, but it was empty. The ACL were setup as above. The /hive znode was accessible using /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM c. The configs are mostly left as default. The most relevant to the issue are here: hive.metastore.kerberos.keytab.file = /etc/security/keytabs/hive.service.keytab hive.metastore.kerberos.principal = hive/_HOST@SOMEREALM hive.metastore.sasl.enabled = true hive.server2.authentication.kerberos.principal = hive/_HOST@SOMEREALM hive.server2.authentication.spnego.keytab = /etc/security/keytabs/spnego.service.keytab hive.server2.authentication.spnego.principal = HTTP/_HOST@SOMEREALM templeton.hive.properties = hive.metastore.local=false,hive.metastore.uris=thrift://<some-address>:9083,hive.metastore.sasl.enabled=true,hive.metastore.execute.setugi=true,hive.metastore.warehouse.dir=/apps/hive/warehouse,hive.exec.mode.local.auto=false,hive.metastore.kerberos.principal=hive/_HOST@SOMEREALM atlas.jaas.KafkaClient.option.principal = hive/_HOST@SOMEREALM hive.llap.zk.sm.principal = hive/_HOST@SOMEREALM hive.llap.daemon.service.principal = hive/_HOST@SOMEREALM xasecure.audit.jaas.Client.option.principal = hive/_HOST@SOMEREALM templeton.kerberos.principal = HTTP/_HOST@SOMEREALM hive.cluster.delegation.token.store.class = org.apache.hadoop.hive.thrift.ZooKeeperTokenStore d. kinit was tried (I expect to try them all, but let me know if some to be double checked): zookeeper user: kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/some_fqdn_nodeA@SOMEREALM kinit -kt /etc/security/keytabs/hive.service.keytab hive/some_fqdn_nodeA@SOMEREALM kinit -kt /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM hive user: kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/some_fqdn_nodeA@SOMEREALM kinit -kt /etc/security/keytabs/hive.service.keytab hive/some_fqdn_nodeA@SOMEREALM kinit -kt /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM e. The error in hivemetastore logs: 2018-01-25 12:30:45,342 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6326)) - org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469) at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6241) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423) at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257) at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160) ... 11 more 2018-01-25 12:30:45,343 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(6159)) - Metastore Thrift Server threw an exception... org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469) at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6241) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688) at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668) at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423) at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257) at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205) at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160) ... 11 more 2018-01-25 12:30:45,395 INFO [Thread-4]: metastore.HiveMetaStore (HiveMetaStore.java:run(6125)) - Shutting down hive metastore. ... View more