If the cluster has only one queue at root level named 'default' and is consuming 100% of the capacity, Ambari will create a queue named 'llap' when HSI is enabled for the 1st time, which is set to (depends on which value is smaller) either : - the minimum required %age for LLAP to work, or - at 20% of cluster's capacity. -------------------------------------------------------------------------------------------------------------------------------------------------------------------- If this is not the case, where there are more than one queue in cluster, user will have to create/set the queue capacity %age in order to be used for LLAP app. Starting with minimum required for queue capacity (shown below), one can increase the queue %age size in order to add up the LLAP nodes in the cluster, as queue size is one of the primary drivers of how many Node Managers nodes will be running LLAP. Reference code for calculating minimum queue size. Following calculations can be a good reference in order to calculate the minimum queue capacity %age to be set by using the following config values as referenced from Ambari UI : - Total Node Manager nodes in Ambari cluster (NMCount). Can be got from Ambari's YARN page. - YARN Node Manager Size (YarnNMSize) (yarn-site/yarn.nodemanager.resource.memory-mb) - YARN minimum container size (YarnMinContSize) (yarn-site/yarn.scheduler.minimum-allocation-mb) - Slider AM container size (SliderAmSize) (hive-interactive-env/slider_am_container_mb). It is calculated as shown here. - Hive Tez Container Size (HiveTezContSize) (hive-interactive-site/hive.tez.container.size) - Tez AM container size (TezAmContSize) (tez-interactive-site/tez.am.resource.memory.mb) NormalizeUp() function is used to normalize the 1st parameter w.r.t. 2nd parameter (YarnMinContSize). Code reference is here, where the snippet function can be used for calculating by putting in a python file and called with correct params, or doing a manual calculation. Min. Total capacity required for queue to run LLAP (MinCapForLlapQueue) = NormalizeUp(SliderAmSize, YarnMinContSize) + NormalizeUp(HiveTezContSize, YarnMinContSize) + NormalizeUp(TezAmContSize, YarnMinContSize) Total Cluster Capacity (ClusterCap) = NMCount * YarnNMSize Min. Queue Percentage Required for queue used for LLAP (in %) (MinQueuePerc) = MinCapForLlapQueue * 100 / ClusterCap Thus, 'MinQueuePerc' value can used to set the queue size to be used for LLAP app. The queue %age can be changed from Ambari > Views > YARN Queue Manager. ... View more

PROBLEM: After upgrade from IOP (v 4.2.5) to HDP (v 2.6.x) in a kerberized setup, HSI start fails with following in HSI start log : WARN impl.LlapZookeeperRegistryImpl: The cluster is not started yet (InvalidACL); will retry ERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCache org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:141) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:99) ~[curator-client-2.7.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache.rebuild(PathChildrenCache.java:323) ~[curator-recipes-2.7.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache.start(PathChildrenCache.java:300) ~[curator-recipes-2.7.1.jar:?] at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.checkPathChildrenCache(LlapZookeeperRegistryImpl.java:827) [hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:790) [hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getInstances(LlapRegistryService.java:139) [hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:579) [hive-llap-server-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:285) [hive-llap-server-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:914) [hive-llap-server-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] ERROR cli.LlapStatusServiceDriver: FAILED: Failed to get instances from llap registry org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver$LlapStatusCliException: Failed to get instances from llap registry at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:581) [hive-llap-server-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:285) [hive-llap-server-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:914) [hive-llap-server-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] Caused by: java.io.IOException: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.checkPathChildrenCache(LlapZookeeperRegistryImpl.java:836) ~[hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:790) ~[hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getInstances(LlapRegistryService.java:139) ~[hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:579) ~[hive-llap-server-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] ... 2 more Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:141) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:99) ~[curator-client-2.7.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache.rebuild(PathChildrenCache.java:323) ~[curator-recipes-2.7.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache.start(PathChildrenCache.java:300) ~[curator-recipes-2.7.1.jar:?] at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.checkPathChildrenCache(LlapZookeeperRegistryImpl.java:827) ~[hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:790) ~[hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getInstances(LlapRegistryService.java:139) ~[hive-exec-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:579) ~[hive-llap-server-2.1.0.2.6.2.0-173.jar:2.1.0.2.6.2.0-173] ... 2 more FAILED: Failed to get instances from llap registry INFO LlapStatusServiceDriverConsole: LLAP status unknown INFO LlapStatusServiceDriverConsole: -------------------------------------------------------------------------------- WARN cli.LlapStatusServiceDriver: Watch mode enabled and got LLAP registry error. Retrying.. WARN impl.LlapZookeeperRegistryImpl: The cluster is not started yet (InvalidACL); will retry Correspondingly, from LLAP YARN application log, we see the following: Caused by: java.io.IOException: Login failure for hive/bug-86157-7.openstacklocal@EXAMPLE.COM from keytab /etc/security/keytabs/hive.llap.zk.sm.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1351) ~[hadoop-common-2.7.3.2.6.2.0-178.jar:?] at org.apache.hadoop.hive.llap.LlapUtil.loginWithKerberos(LlapUtil.java:78) ~[hive-exec-2.1.0.2.6.2.0-178.jar:2.1.0.2.6.2.0-178] at org.apache.hadoop.hive.llap.security.SecretManager.createLlapZkConf(SecretManager.java:202) ~[hive-exec-2.1.0.2.6.2.0-178.jar:2.1.0.2.6.2.0-178] ... 4 more Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897) ~[?:1.8.0_141] at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) ~[?:1.8.0_141] at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[?:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141] at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) ~[?:1.8.0_141] at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ~[?:1.8.0_141] at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ~[?:1.8.0_141] at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ~[?:1.8.0_141] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_141] at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ~[?:1.8.0_141] at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ~[?:1.8.0_141] at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1340) ~[hadoop-common-2.7.3.2.6.2.0-178.jar:?] at org.apache.hadoop.hive.llap.LlapUtil.loginWithKerberos(LlapUtil.java:78) ~[hive-exec-2.1.0.2.6.2.0-178.jar:2.1.0.2.6.2.0-178] at org.apache.hadoop.hive.llap.security.SecretManager.createLlapZkConf(SecretManager.java:202) ~[hive-exec-2.1.0.2.6.2.0-178.jar:2.1.0.2.6.2.0-178] REASON : After IOP-HDP upgrade, the keytab file permissions for "hive.llap.zk.sm.keytab" is only owner readable. # ls -al /etc/security/keytabs/hive.llap.zk.sm.keytab -r--------. 1 yarn hadoop 428 Aug 15 19:34 hive.llap.zk.sm.keytab Thus, hive user is not able to access hive.llap.zk.sm.keytab. The group not having read permission is because IOP kerberos.json file for YARN have permissions removed for it. FIX: Regenerating keytabs via Ambari fixes the permission issue, as hive user can now access the keytabs being part of hadoop group. # ls -al /etc/security/keytabs/hive.llap.zk.sm.keytab -r--r-----. 1 yarn hadoop 428 Aug 15 20:26 hive.llap.zk.sm.keytab - Once regeneration is done, start HSI again. P.S. : Hive Server Interactive is made up of 2 sub-components : LLAP and Hive2/HiveServer2. ... View more

PROBLEM: After upgrade from IOP (v 4.2.0) to HDP (v 2.6.x) in a kerberized setup, HSI install fails with Configuration parameter 'hive.llap.daemon.service.principal' missing in dictionary. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 616, in <module> HiveServerInteractive().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 75, in install import params File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params.py", line 27, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py", line 697, in <module> hive_llap_principal = (config['configurations']['hive-interactive-site']['hive.llap.daemon.service.principal']).replace('_HOST',hostname.lower()) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'hive.llap.daemon.service.principal' was not found in configurations dictionary! Reason: HSI related keytabs are missing on cluster. Fix: Regenerate keytabs via Ambari. - Once regeneration is done, you can reinstall HSI and start it. P.S. : Hive Server Interactive is made up of 2 sub-components : LLAP and Hive2/HiveServer2. Thanks to @vrathod for reporting it. ... View more

PROBLEM: After upgrade from IOP (v 4.2.0) to HDP (v 2.6.x) in a kerberized setup, HSI install fails with Configuration parameter 'hive.llap.daemon.service.principal' missing in dictionary. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 616, in <module> HiveServerInteractive().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 75, in install import params File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params.py", line 27, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py", line 697, in <module> hive_llap_principal = (config['configurations']['hive-interactive-site']['hive.llap.daemon.service.principal']).replace('_HOST',hostname.lower()) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'hive.llap.daemon.service.principal' was not found in configurations dictionary! Reason: HSI related keytabs are missing on cluster. Fix: Regenerate keytabs via Ambari. - Once regeneration is done, you can reinstall HSI and start it. P.S. : Hive Server Interactive is made up of 2 sub-components : LLAP and Hive2/HiveServer2. ... View more

Issue: HSI's (Tech Preview) component LLAP start fails in kerberized setup because of missing keytabs. When HSI is started, its component LLAP fails with below trace: INFO impl.LlapRegistryService: Using LLAP registry (client) type: Service LlapRegistryService in state LlapRegistryService: STARTED INFO state.ConnectionStateManager: State change: CONNECTED ERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCache. Exception: {} org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.5.3.0-37.jar:3.4.6-37--1] at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:141) ~[curator-client- 2.7.1.jar:?] at org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:99) ~[curator-client-2.7.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache.rebuild(PathChildrenCache.java:323) ~[curator-recipes-2.7.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache.start(PathChildrenCache.java:300) ~[curator-recipes-2.7.1.jar:?] at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.checkPathChildrenCache(LlapZookeeperRegistryImpl.java:757) [hive-exec-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:725) [hive-exec-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getInstances(LlapRegistryService.java:129) [hive-exec-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:490) [hive-llap-server-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:245) [hive-llap-server-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:941) [hive-llap-server-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] ERROR cli.LlapStatusServiceDriver: FAILED: Failed to get instances from llap registry org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver$LlapStatusCliException: Failed to get instances from llap registry at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:492) [hive-llap-server-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:245) [hive-llap-server-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:941) [hive-llap-server-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] Caused by: java.io.IOException: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.checkPathChildrenCache(LlapZookeeperRegistryImpl.java:760) ~[hive-exec-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:725) ~[hive-exec-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getInstances(LlapRegistryService.java:129) ~[hive-exec-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.populateAppStatusFromLlapRegistry(LlapStatusServiceDriver.java:490) ~[hive-llap-server-2.1.0.2.5.3.0-37.jar:2.1.0.2.5.3.0-37] ... 2 more This can happen in case the HSI is enabled after kerberizing the cluster. Reason: - This is because HSI needs 2 ketab files : 'hive.service.keytab' and 'hive.llap.zk.sm.keytab' present on all the YARN's NodeManager nodes. - If HSI is not enabled before the cluster's kerberization, the above two keytab files will not get distributed on all the NodeManager nodes, unlike when HSI is Enabled before kerberization. Thus, the error: Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive  at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1] because the ZK node is not created / missing. zk: localhost:2181(CONNECTED) 3] ls /llap-sasl [] zk node is missing Resolution: - Regenerating keytabs from Ambari Kerberos page, will distribute the above keytab files on all NodeManager Nodes. - Further, do confirm that Hive's config hive.llap.zk.sm.connectionString is updated with the list of all Zookeeper Nodes in the cluster. For example: zk.host1.org:2181,zk.host2.org:2181,zk.host3.org:2181 The Zookeeper Nodes list ca be got from here: Note to append the Port Numbers as mentioned in example. Restart HSI to confirm the behavior. ... View more