Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Cloudera Community

Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.
Solved Go to solution

Hive Metastore won't start after enabling Kerberos. Znodes are not created

Labels:
avatar
author-rank sergejs_andreje
New Member

Created ‎01-26-2018 11:43 AM

I have faced with similar error as here: https://community.hortonworks.com/questions/28589/hive-metastore-wont-start-after-enabling-kerberos.... (due to message size limitations, couldn't comment there)

Cluster layout:
NodeA - majority of hadoop services

NodeB - hadoop clients installed. Kerberos installed.

Stack:

OS:

Kerberos 5 version 1.15.1

Ambari 2.4.1.0

HDP: 2.5.0.0

-- hive 1.2.1.2.5

-- zookeeper 3.4.6.2.5

-- kerberos 1.10.3-10

How to reproduce:

1. Unkerberized cluster is deployed it works fine.

2. I kerberize the cluster: all services are up, except hive metastore (shown as start is successful, but fails immediately after start)

Additional info:

a. If I unkeberize the cluster - it works fine again.

b. In zookeeper there is not created even /hive znode. The links above were checked and when I have added property "hive.cluster.delegation.token.store.zookeeper.acl=sasl:hive:cdrwa"

then (after the change) I restarted the hive services: /hive znode was created with named properties, but it was empty. The ACL were setup as above. The /hive znode was accessible using /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM

c. The configs are mostly left as default. The most relevant to the issue are here:

hive.metastore.kerberos.keytab.file = /etc/security/keytabs/hive.service.keytab
hive.metastore.kerberos.principal = hive/_HOST@SOMEREALM
hive.metastore.sasl.enabled = true
hive.server2.authentication.kerberos.principal = hive/_HOST@SOMEREALM
hive.server2.authentication.spnego.keytab = /etc/security/keytabs/spnego.service.keytab
hive.server2.authentication.spnego.principal = HTTP/_HOST@SOMEREALM
   
templeton.hive.properties = hive.metastore.local=false,hive.metastore.uris=thrift://<some-address>:9083,hive.metastore.sasl.enabled=true,hive.metastore.execute.setugi=true,hive.metastore.warehouse.dir=/apps/hive/warehouse,hive.exec.mode.local.auto=false,hive.metastore.kerberos.principal=hive/_HOST@SOMEREALM
atlas.jaas.KafkaClient.option.principal = hive/_HOST@SOMEREALM
   
hive.llap.zk.sm.principal = hive/_HOST@SOMEREALM
hive.llap.daemon.service.principal = hive/_HOST@SOMEREALM
   
xasecure.audit.jaas.Client.option.principal = hive/_HOST@SOMEREALM
templeton.kerberos.principal = HTTP/_HOST@SOMEREALM
   
hive.cluster.delegation.token.store.class = org.apache.hadoop.hive.thrift.ZooKeeperTokenStore

d. kinit was tried (I expect to try them all, but let me know if some to be double checked):

zookeeper user:
kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.service.keytab hive/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM

hive user:
kinit -kt /etc/security/keytabs/zk.service.keytab zookeeper/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.service.keytab hive/some_fqdn_nodeA@SOMEREALM
kinit -kt /etc/security/keytabs/hive.llap.zk.sm.keytab hive/some_fqdn_nodeA@SOMEREALM

e. The error in hivemetastore logs:

2018-01-25 12:30:45,342 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6326)) - org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6241)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
        ... 11 more

2018-01-25 12:30:45,343 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(6159)) - Metastore Thrift Server threw an exception...
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationMETASTORE/keys
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6241)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hive/cluster/delegationMETASTORE/keys
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
        ... 11 more 

2018-01-25 12:30:45,395 INFO [Thread-4]: metastore.HiveMetaStore (HiveMetaStore.java:run(6125)) - Shutting down hive metastore.
9,300 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank sergejs_andreje
New Member

Created ‎01-30-2018 09:55 AM

The issue looks to be within check of kerberos tickets: HiveMetastore wasn't using them.

Installed HDP2.5.3.0+ with the same configs and it worked.

View solution in original post

8,602 Views
0 Kudos
3 REPLIES 3

avatar
author-rank jsensharma author-rank
Master Mentor

Created on ‎01-26-2018 11:48 AM - edited ‎08-17-2019 09:53 PM

@Sergejs Andrejevs

Can you please check if your "hive-site.xml" has the following property defined and set to "true". I see it in your "templeton.hive.properties" but please verify your hive-site.xml as well.

hive.metastore.sasl.enabled =  true

.

Ambari UI --> Hive --> Configs --> Advanced --> Advanced hive-site

And then in the above location please check if the property "hive.metastore.sasl.enabled" is set or not? If not then please try setting it and followed by Hive Service restart.

56479-hive-sasl.png

.

8,602 Views
0 Kudos

avatar
author-rank sergejs_andreje
New Member

Created ‎01-26-2018 12:07 PM

hive.metastore.sasl.enabled = true

I'll update the initial post too with this info. Thanks for noting.

8,602 Views
0 Kudos

avatar
author-rank sergejs_andreje
New Member

Created ‎01-30-2018 09:55 AM

The issue looks to be within check of kerberos tickets: HiveMetastore wasn't using them.

Installed HDP2.5.3.0+ with the same configs and it worked.

8,603 Views
0 Kudos
Announcements
What's New @ Cloudera
Announcing Cloudera Data Flow 3.2 for Cloudera on Cloud
Community Announcements
Community Upgrade Announcement
What's New @ Cloudera
Announcing Cloudera Flow Management Operator for Kubernetes ...
What's New @ Cloudera
Announcing Cloudera Streams Messaging Operator for Kubernete...
Community Announcements
July 2026 Community Highlights
View More Announcements