Created 09-07-2016 06:37 PM
On fresh installed HDP-2.5 I can’t start HiveServer2 Interactive. Cluster is High Available. I tried to install HiveServer2 Interactive on both ActiveNN and StandbyNN, but with the same unsuccessful result. I didn't find any obvious exeptions in logs.
Here stderr:
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 512, in check_llap_app_status status = do_retries() File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py", line 55, in wrapper return function(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 505, in do_retries raise Fail(status_str) Fail: LLAP app 'llap0' current state is COMPLETE. 2016-09-07 20:37:48,705 - LLAP app 'llap0' deployment unsuccessful. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 535, in <module> HiveServerInteractive().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 123, in start raise Fail("Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.") resource_management.core.exceptions.Fail: Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.
sdtout too long, so here some some excerpts:
2016-09-07 20:31:49,638 - Starting LLAP 2016-09-07 20:31:49,643 - Command: /usr/hdp/current/hive-server2-hive2/bin/hive --service llap --instances 1 --slider-am-container-mb 5120 --size 30720m --cache 0m --xmx 29696m --loglevel INFO --output /var/lib/ambari-agent/tmp/llap-slider2016-09-07_17-31-49 --args " -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:TLABSize=8m -XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts -XX:MetaspaceSize=1024m -XX:InitiatingHeapOccupancyPercent=80 -XX:MaxGCPauseMillis=200" 2016-09-07 20:31:49,643 - checked_call['/usr/hdp/current/hive-server2-hive2/bin/hive --service llap --instances 1 --slider-am-container-mb 5120 --size 30720m --cache 0m --xmx 29696m --loglevel INFO --output /var/lib/ambari-agent/tmp/llap-slider2016-09-07_17-31-49 --args " -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:TLABSize=8m -XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts -XX:MetaspaceSize=1024m -XX:InitiatingHeapOccupancyPercent=80 -XX:MaxGCPauseMillis=200"'] {'logoutput': True, 'user': 'hive', 'stderr': -1} which: no hbase in (/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] INFO cli.LlapServiceDriver: LLAP service driver invoked with arguments=--hiveconf INFO conf.HiveConf: Found configuration file file:/etc/hive2/2.5.0.0-1245/0/conf.server/hive-site.xml WARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist WARN cli.LlapServiceDriver: Ignoring unknown llap server parameter: [hive.aux.jars.path] WARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist INFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore INFO metastore.ObjectStore: ObjectStore, initialize called WARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,Database,Type,FieldSchema,Order" INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL INFO metastore.ObjectStore: Initialized ObjectStore INFO metastore.HiveMetaStore: Added admin role in metastore INFO metastore.HiveMetaStore: Added public role in metastore INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty INFO metastore.HiveMetaStore: 0: get_all_functions INFO HiveMetaStore.audit: ugi=hive ip=unknown-ip-addr cmd=get_all_functions WARN cli.LlapServiceDriver: Java versions might not match : JAVA_HOME=[/usr/jdk64/jdk1.8.0_77],process jre=[/usr/jdk64/jdk1.8.0_77/jre] INFO cli.LlapServiceDriver: Using [/usr/jdk64/jdk1.8.0_77] for JAVA_HOME INFO cli.LlapServiceDriver: Copied hadoop metrics2 properties file from file:/etc/hive2/2.5.0.0-1245/0/conf.server/hadoop-metrics2-llapdaemon.properties INFO cli.LlapServiceDriver: LLAP service driver finished Prepared /var/lib/ambari-agent/tmp/llap-slider2016-09-07_17-31-49/run.sh for running LLAP on Slider 2016-09-07 20:32:18,650 - checked_call returned (0, 'Prepared /var/lib/ambari-agent/tmp/llap-slider2016-09-07_17-31-49/run.sh for running LLAP on Slider', 'which: no hbase in (/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent)\nSLF4J: Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\nSLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]\nINFO cli.LlapServiceDriver: LLAP service driver invoked with arguments=--hiveconf\nINFO conf.HiveConf: Found configuration file file:/etc/hive2/2.5.0.0-1245/0/conf.server/hive-site.xml\nWARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist\nWARN cli.LlapServiceDriver: Ignoring unknown llap server parameter: [hive.aux.jars.path]\nWARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist\nINFO metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore\nINFO metastore.ObjectStore: ObjectStore, initialize called\nWARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist\nINFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,Database,Type,FieldSchema,Order"\nINFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL\nINFO metastore.ObjectStore: Initialized ObjectStore\nINFO metastore.HiveMetaStore: Added admin role in metastore\nINFO metastore.HiveMetaStore: Added public role in metastore\nINFO metastore.HiveMetaStore: No user is added in admin role, since config is empty\nINFO metastore.HiveMetaStore: 0: get_all_functions\nINFO HiveMetaStore.audit: ugi=hive\tip=unknown-ip-addr\tcmd=get_all_functions\t\nWARN cli.LlapServiceDriver: Java versions might not match : JAVA_HOME=[/usr/jdk64/jdk1.8.0_77],process jre=[/usr/jdk64/jdk1.8.0_77/jre]\nINFO cli.LlapServiceDriver: Using [/usr/jdk64/jdk1.8.0_77] for JAVA_HOME\nINFO cli.LlapServiceDriver: Copied hadoop metrics2 properties file from file:/etc/hive2/2.5.0.0-1245/0/conf.server/hadoop-metrics2-llapdaemon.properties\nINFO cli.LlapServiceDriver: LLAP service driver finished') 2016-09-07 20:32:18,651 - Run file path: /var/lib/ambari-agent/tmp/llap-slider2016-09-07_17-31-49/run.sh 2016-09-07 20:32:18,652 - Execute['/var/lib/ambari-agent/tmp/llap-slider2016-09-07_17-31-49/run.sh'] {'user': 'hive'} 2016-09-07 20:32:48,625 - Submitted LLAP app name : llap0 2016-09-07 20:32:48,627 - checked_call['/usr/hdp/current/hive-server2-hive2/bin/hive --service llapstatus --name llap0 --findAppTimeout 0'] {'logoutput': False, 'user': 'hive', 'stderr': -1} 2016-09-07 20:32:59,607 - checked_call returned (0, '{\n "amInfo" : {\n "appName" : "llap0",\n "appType" : "org-apache-slider",\n "appId" : "application_1473264739795_0004"\n },\n "state" : "LAUNCHING",\n "appStartTime" : 1473269567664\n}', 'which: no hbase in (/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent)\nSLF4J: Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\nSLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]\nINFO cli.LlapStatusServiceDriver: LLAP status invoked with arguments = --hiveconf\nINFO conf.HiveConf: Found configuration file file:/etc/hive2/2.5.0.0-1245/0/conf.server/hive-site.xml\nWARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist\nINFO impl.TimelineClientImpl: Timeline service address: http://hdp-nn1.co.vectis.local:8188/ws/v1/timeline/\nINFO client.AHSProxy: Connecting to Application History server at hdp-nn1.co.vectis.local/10.255.242.180:10200\nINFO cli.LlapStatusServiceDriver: LLAP status finished') 2016-09-07 20:32:59,608 - Received 'llapstatus' command 'output' : { "amInfo" : { "appName" : "llap0", "appType" : "org-apache-slider", "appId" : "application_1473264739795_0004" }, "state" : "LAUNCHING", "appStartTime" : 1473269567664 } 2016-09-07 20:32:59,608 - Marker index for start of JSON data for 'llapsrtatus' comamnd : 0 2016-09-07 20:32:59,610 - LLAP app 'llap0' current state is LAUNCHING. 2016-09-07 20:32:59,611 - Will retry 19 time(s), caught exception: LLAP app 'llap0' current state is LAUNCHING.. Sleeping for 2 sec(s) 2016-09-07 20:33:01,614 - checked_call['/usr/hdp/current/hive-server2-hive2/bin/hive --service llapstatus --name llap0 --findAppTimeout 0'] {'logoutput': False, 'user': 'hive', 'stderr': -1} 2016-09-07 20:33:15,295 - checked_call returned (0, '{\n "amInfo" : {\n "appName" : "llap0",\n "appType" : "org-apache-slider",\n "appId" : "application_1473264739795_0004",\n "containerId" : "container_e12_1473264739795_0004_01_000001",\n "hostname" : "hdp-dn2.co.vectis.local",\n "amWebUrl" : "http://hdp-dn2.co.vectis.local:40485/"\n },\n "state" : "LAUNCHING",\n "originalConfigurationPath" : "hdfs://prodcluster/user/hive/.slider/cluster/llap0/snapshot",\n "generatedConfigurationPath" : "hdfs://prodcluster/user/hive/.slider/cluster/llap0/generated",\n "desiredInstances" : 1,\n "liveInstances" : 0,\n "appStartTime" : 1473269583908\n}', 'which: no hbase in (/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent)\nSLF4J: Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: Found binding in [jar:file:/usr/hdp/2.5.0.0-1245/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\nSLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]\nINFO cli.LlapStatusServiceDriver: LLAP status invoked with arguments = --hiveconf\nINFO conf.HiveConf: Found configuration file file:/etc/hive2/2.5.0.0-1245/0/conf.server/hive-site.xml\nWARN conf.HiveConf: HiveConf of name hive.llap.daemon.allow.permanent.fns does not exist\nINFO impl.TimelineClientImpl: Timeline service address: http://hdp-nn1.co.vectis.local:8188/ws/v1/timeline/\nINFO client.AHSProxy: Connecting to Application History server at hdp-nn1.co.vectis.local/10.255.242.180:10200\nWARN curator.CuratorZookeeperClient: session timeout [10000] is less than connection timeout [15000]\nINFO impl.LlapZookeeperRegistryImpl: Llap Zookeeper Registry is enabled with registryid: llap0\nINFO impl.LlapRegistryService: Using LLAP registry type org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl@4e6f2bb5\nINFO impl.LlapZookeeperRegistryImpl: UGI security is not enabled, or non-daemon environment. Skipping setting up ZK auth.\nINFO imps.CuratorFrameworkImpl: Starting\nINFO impl.LlapRegistryService: Using LLAP registry (client) type: Service LlapRegistryService in state LlapRegistryService: STARTED\nINFO state.ConnectionStateManager: State change: CONNECTED\nINFO cli.LlapStatusServiceDriver: No information found in the LLAP registry\nINFO cli.LlapStatusServiceDriver: LLAP status finished') 2016-09-07 20:33:15,295 - Received 'llapstatus' command 'output' : { "amInfo" : { "appName" : "llap0", "appType" : "org-apache-slider", "appId" : "application_1473264739795_0004", "containerId" : "container_e12_1473264739795_0004_01_000001", "hostname" : "hdp-dn2.co.vectis.local", "amWebUrl" : "http://hdp-dn2.co.vectis.local:40485/" }, "state" : "LAUNCHING", "originalConfigurationPath" : "hdfs://prodcluster/user/hive/.slider/cluster/llap0/snapshot", "generatedConfigurationPath" : "hdfs://prodcluster/user/hive/.slider/cluster/llap0/generated", "desiredInstances" : 1, "liveInstances" : 0, "appStartTime" : 1473269583908 }
Created 09-12-2016 09:44 AM
Solved it! The problem was with the parameters:
hive.llap.daemon.yarn.container.mb llap_heap_size
Ambari sets default value of llap_heap_size about 96% of hive.llap.daemon.yarn.container.mb (when I move slider "% of Cluster Capacity"), although it should be about 80%. Manual setting the correct parameters allowed to start the HiveServer2 Interactive.
Created 09-12-2016 09:44 AM
Solved it! The problem was with the parameters:
hive.llap.daemon.yarn.container.mb llap_heap_size
Ambari sets default value of llap_heap_size about 96% of hive.llap.daemon.yarn.container.mb (when I move slider "% of Cluster Capacity"), although it should be about 80%. Manual setting the correct parameters allowed to start the HiveServer2 Interactive.
Created 11-22-2016 07:43 AM
Hi @Alena Melnikova How do you solve it? I had started once , but failed others attempts with the same
"current state is LAUNCHING"
Created 04-19-2017 07:05 PM
@Alena Melnikova, @Huahua Wei - i'm getting the same issue, and On my cluster, llap_heap_size is ~80% hive.llap.daemon.yarn.container.mb, but HiveServer Interactive is not starting up. Any ideas on what else needs to be done ?
------error ---------
----------------------
Created 06-09-2017 08:28 AM
you can try to update your openssl
Created 11-23-2016 08:24 PM
The value of hive.llap.daemon.yarn.container.mb can be found in /etc/hive2/conf/hive-site.xml. Or in Ambari Hive -> Config -> Settings -> Interactive Query -> "Memory per daemon". The "LLAP heap size" can be modified in Ambari under the "Advanced hive-interactive-env" section.
Created 10-14-2016 01:25 AM
I'm also having same the problem.
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 512, in check_llap_app_status status = do_retries() File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py", line 55, in wrapper return function(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 505, in do_retries raise Fail(status_str) Fail: LLAP app 'llap0' current state is LAUNCHING. 2016-10-14 08:16:01,061 - LLAP app 'llap0' deployment unsuccessful. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 535, in <module> HiveServerInteractive().execute() File "/usr/lib/python2.6/site-packages/resouroutput-702.txtce_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 123, in start raise Fail("Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.") resource_management.core.exceptions.Fail: Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED. ~
Created 06-09-2017 08:27 AM
if nothing worked , you can try update your openssl
Created 06-10-2017 02:39 PM
I have updated openssl and made the changes to both. It is still not working does anyone have any other ideas on getting HIve Interactive with LLAP working?
Created 03-13-2018 02:30 PM
I've the same issue. Any one can advice?