Support Questions
Find answers, ask questions, and share your expertise

I am not able to start Hiveserver2 interactive with exception FAILED: Failed to get instances from llap registry

I am getting below exception ERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCache. Exception: {}

Fail: Execution of '/usr/hdp/current/hive-server2-hive2/bin/hive --service llapstatus --name llap0 --findAppTimeout 0' returned 40. SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
INFO cli.LlapStatusServiceDriver: LLAP status invoked with arguments = --hiveconf
INFO conf.HiveConf: Found configuration file file:/etc/hive2/2.5.0.0-1245/0/conf.server/hive-site.xml
WARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist
WARN conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value
INFO tools.SliderUtils: JVM initialized into secure mode with kerberos realm IHGEXT.GLOBAL
INFO impl.TimelineClientImpl: Timeline service address: http://mksd1dlvisbm014.c.iron-potion-771.internal:8188/ws/v1/timeline/
INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.5.5-hadoop2
INFO client.AHSProxy: Connecting to Application History server at mksd1dlvisbm014.c.iron-potion-771.internal/10.216.24.98:10200
WARN curator.CuratorZookeeperClient: session timeout [10000] is less than connection timeout [15000]
INFO impl.LlapZookeeperRegistryImpl: Llap Zookeeper Registry is enabled with registryid: llap0
INFO impl.LlapRegistryService: Using LLAP registry type org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl@12df28be
INFO impl.LlapZookeeperRegistryImpl: UGI security is not enabled, or non-daemon environment. Skipping setting up ZK auth.
INFO imps.CuratorFrameworkImpl: Starting
INFO impl.LlapRegistryService: Using LLAP registry (client) type: Service LlapRegistryService in state LlapRegistryService: STARTED
INFO state.ConnectionStateManager: State change: CONNECTED
ERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCache. Exception: {}
org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1]
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1]
	at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:141) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:99) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache.rebuild(PathChildrenCache.java:323) ~[curator-recipes-2.7.1.jar:?]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache.start(PathChildrenCache.java:300) ~[curator-recipes-2.7.1.jar:?]
	at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.checkPathChildrenCache(LlapZookeeperRegistryImpl.java:679) [hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:646) [hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getInstances(LlapRegistryService.java:129) [hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:489) [hive-llap-server-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:244) [hive-llap-server-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:944) [hive-llap-server-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
ERROR cli.LlapStatusServiceDriver: FAILED: Failed to get instances from llap registry
org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver$LlapStatusCliException: Failed to get instances from llap registry
	at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:491) [hive-llap-server-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:244) [hive-llap-server-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:944) [hive-llap-server-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
Caused by: java.io.IOException: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive
	at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.checkPathChildrenCache(LlapZookeeperRegistryImpl.java:682) ~[hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:646) ~[hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getInstances(LlapRegistryService.java:129) ~[hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:489) ~[hive-llap-server-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	... 2 more
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1]
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1]
	at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:141) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:99) ~[curator-client-2.7.1.jar:?]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache.rebuild(PathChildrenCache.java:323) ~[curator-recipes-2.7.1.jar:?]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache.start(PathChildrenCache.java:300) ~[curator-recipes-2.7.1.jar:?]
	at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.checkPathChildrenCache(LlapZookeeperRegistryImpl.java:679) ~[hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:646) ~[hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getInstances(LlapRegistryService.java:129) ~[hive-exec-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:489) ~[hive-llap-server-2.1.0.2.5.0.0-1245.jar:2.1.0.2.5.0.0-1245]
	... 2 more
FAILED: Failed to get instances from llap registry
INFO cli.LlapStatusServiceDriver: LLAP status finished
2017-02-09 15:11:44,756 - LLAP app 'llap0' deployment unsuccessful.
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 535, in <module>
    HiveServerInteractive().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 123, in start
    raise Fail("Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.")
resource_management.core.exceptions.Fail: Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED
2 REPLIES 2

New Contributor

Getting the same error.

I have resolved by following below steps:

Error seems to be hive is not able to create required directories in zookeeper so need to create the following directories in Znode

./zkCli.sh -server mksd1dlvisbm013.c.iron-potion-771.internal:2181

We manually created necessary entries:

(CONNECTED) 62] create /zkdtsm_hive_llap0 “” world:anyone:cdrwa

(CONNECTED) 63] create /zkdtsm_hive_llap0/ZKDTSMRoot "" world:anyone:cdrwa

(CONNECTED) 64] create /zkdtsm_hive_llap0/ZKDTSMRoot/ZKDTSMKeyIdRoot "" sasl:hive:cdrwa

(CONNECTED) 65] create /zkdtsm_hive_llap0/ZKDTSMRoot/ZKDTSMMasterKeyRoot "" sasl:hive:cdrwa

(CONNECTED) 66] create /zkdtsm_hive_llap0/ZKDTSMRoot/ZKDTSMTokensRoot "" sasl:hive:cdrwa

(CONNECTED) 67] create /zkdtsm_hive_llap0/ZKDTSMRoot/ZKDTSMSeqNumRoot "" sasl:hive:cdrwa

After this, llap is not starting up complaining about insufficient hive.llap.io.memory.size and we increased the same to 512MB

Second option would be bring down the zookeeper and reinstall it again as something might causing the issue in zookeeper to not allowing the directory creation by hive