Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HiveServer2 Interactive Start failed

avatar
Contributor

Hello I tried to activate LLAP and add HiveServer2 Interactive service but the service fails to start

I could not find any solution, any idea?

here is part of output in ambari when process is starting:

2017-04-28 15:45:34,633 [main] INFO  util.ExitUtil - Exiting with status 0
2017-04-28 15:45:35,491 - Submitted LLAP app name : llap0
2017-04-28 15:45:35,492 - 





2017-04-28 15:45:35,492 - LLAP status command : /usr/hdp/current/hive-server2-hive2/bin/hive --service llapstatus -w -r 0.8 -i 2 -t 400
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARN conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value

LLAPSTATUS WatchMode with timeout=400 s
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances
--------------------------------------------------------------------------------
WARN cli.LlapStatusServiceDriver: Application stopped while launching. COMPLETE state reached while waiting for RUNNING state. Failing fast..
LLAP Application already complete. ApplicationId=application_1493310509760_0013
	FAILED container:  container_e23_1493310509760_0013_01_000003, Logs at: http://somehost1:19888/jobhistory/logs/somehostN1:45454/container_e23_1493310509760_0013_01_000003/c...
	FAILED container:  container_e23_1493310509760_0013_01_000002, Logs at: http://somehost1:19888/jobhistory/logs/somehostN2:45454/container_e23_1493310509760_0013_01_000002/c...
	FAILED container:  container_e23_1493310509760_0013_01_000005, Logs at: http://somehost1:19888/jobhistory/logs/somehostN3:45454/container_e23_1493310509760_0013_01_000005/c...
	FAILED container:  container_e23_1493310509760_0013_01_000004, Logs at: http://somehost1:19888/jobhistory/logs/somehostN4:45454/container_e23_1493310509760_0013_01_000004/c...
	FAILED container:  container_e23_1493310509760_0013_01_000007, Logs at: http://somehost1:19888/jobhistory/logs/somehostN5:45454/container_e23_1493310509760_0013_01_000007/c...
	FAILED container:  container_e23_1493310509760_0013_01_000006, Logs at: http://somehost1:19888/jobhistory/logs/somehostN6:45454/container_e23_1493310509760_0013_01_000006/c...
	FAILED container:  container_e23_1493310509760_0013_01_000009, Logs at: http://somehost1:19888/jobhistory/logs/---z---:45454/container_e23_1493310509760_0013_01_000009/ctx/...
	FAILED container:  container_e23_1493310509760_0013_01_000008, Logs at: http://somehost1:19888/jobhistory/logs/----z---:45454/container_e23_1493310509760_0013_01_000008/ctx...
	FAILED container:  container_e23_1493310509760_0013_01_000010, Logs at: http://somehost1:19888/jobhistory/logs/-----z----:45454/container_e23_1493310509760_0013_01_000010/c...
Unstable Application Instance : - failed with component LLAP failed 'recently' 6 times (6 in startup); threshold is 5 - last failure: Failure container_e23_1493310509760_0013_01_000007 on host somehostN12 (0): http://somehost1:19888/jobhistory/logs/somehostN12:45454/container_e23_1493310509760_0013_01_000007/...
--------------------------------------------------------------------------------
P.S. not real hostnames, I've deleted or changed
1 ACCEPTED SOLUTION

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
8 REPLIES 8

avatar

As the error message suggests, the containers launched by LLAP have failed, can you check the failed application logs?

avatar
Contributor
 Log Type: llap-daemon-hive-somehostname.log


Log Upload Time: Mon May 01 13:04:54 +0400 2017


Log Length: 16486


Showing 4096 bytes of 16486 total. Click here for the full log.


.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
2017-05-01T13:04:23,578 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception
java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
Caused by: java.lang.IllegalArgumentException
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
2017-05-01T13:04:23,578 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked
2017-05-01T13:04:23,579 WARN  [main ()] org.apache.hadoop.metrics2.util.MBeans: Error unregistering Hadoop:service=LlapDaemon,name=LlapDaemonInfo
javax.management.InstanceNotFoundException: Hadoop:service=LlapDaemon,name=LlapDaemonInfo
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) ~[?:1.8.0_112]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427) ~[?:1.8.0_112]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415) ~[?:1.8.0_112]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) ~[?:1.8.0_112]
	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:110) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.shutdown(LlapDaemon.java:445) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:537) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]

avatar
Contributor

and this is full log of LLAP:

Log Type: llap-daemon-hive-somehost5.somedomain.log


Log Upload Time: Mon May 01 13:04:54 +0400 2017


Log Length: 16486


2017-05-01T13:04:21,632 INFO  [main ()] org.apache.hadoop.hive.conf.HiveConf: Found configuration file file:/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032/container_e23_1493310509760_0032_01_000007/app/install/conf/hive-site.xml
2017-05-01T13:04:22,044 INFO  [main ()] org.apache.hadoop.hive.llap.LlapUtil: Using local dirs from environment: /DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032
2017-05-01T13:04:22,200 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LLAP daemon logging initialized from file:/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032/container_e23_1493310509760_0032_01_000007/app/install/conf/llap-daemon-log4j2.properties in 153 ms. Async: true
2017-05-01T13:04:22,202 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: 


$$\       $$\        $$$$$$\  $$$$$$$\
$$ |      $$ |      $$  __$$\ $$  __$$\
$$ |      $$ |      $$ /  $$ |$$ |  $$ |
$$ |      $$ |      $$$$$$$$ |$$$$$$$  |
$$ |      $$ |      $$  __$$ |$$  ____/
$$ |      $$ |      $$ |  $$ |$$ |
$$$$$$$$\ $$$$$$$$\ $$ |  $$ |$$ |
\________|\________|\__|  \__|\__|




2017-05-01T13:04:22,203 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Zookeeper Quorum: somehost1.somedomain:2181,somehost2.somedomain:2181,somehost3.somedomain:2181
2017-05-01T13:04:22,462 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Starting daemon as user: hive
2017-05-01T13:04:22,475 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Attempting to start LlapDaemonConf with the following configuration: maxJvmMemory=839909370 (801.00MB), requestedExecutorMemory=838860800 (800.00MB), llapIoCacheSize=134217728 (128.00MB), xmxHeadRoomMemory=41995468 (40.05MB), adjustedExecutorMemory=796865332 (759.95MB), numExecutors=1, llapIoEnabled=true, llapIoCacheIsDirect=true, rpcListenerPort=0, mngListenerPort=15004, webPort=15002, outputFormatSvcPort=15003, workDirs=[/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032], shufflePort=15551, waitQueueSize= 10, enablePreemption= true
2017-05-01T13:04:22,546 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2-llapdaemon.properties
2017-05-01T13:04:22,800 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink.
2017-05-01T13:04:22,805 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Identified hostname = somehost5.somedomain, serviceName = llapdaemon
2017-05-01T13:04:22,870 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.availability.MetricSinkWriteShardHostnameHashingStrategy: Calculated collector shard somehost3.somedomain based on hostname: somehost5.somedomain
2017-05-01T13:04:22,871 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Collector Uri: http://somehost3.somedomain:6188/ws/v1/timeline/metrics
2017-05-01T13:04:22,871 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Container Metrics Uri: http://somehost3.somedomain:6188/ws/v1/timeline/containermetrics
2017-05-01T13:04:22,884 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink timeline started
2017-05-01T13:04:22,903 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2017-05-01T13:04:22,904 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system started
2017-05-01T13:04:22,909 INFO  [org.apache.hadoop.util.JvmPauseMonitor$Monitor@324a0017 ()] org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
2017-05-01T13:04:22,940 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Started LlapMetricsSystem with displayName: LlapDaemonExecutorMetrics-somehost5.somedomain sessionId: 7b7897de-b8a8-48d5-b4a8-1143f5ef91ea
2017-05-01T13:04:22,954 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.AMReporter: Setting up AMReporter with heartbeatInterval(ms)=10000, retryTime(ms)=10000, retrySleep(ms)=2000
2017-05-01T13:04:22,966 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl: Creating: LlapProtocolServerImpl with port configured to: 0
2017-05-01T13:04:23,131 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.QueryTracker: QueryTracker setup with numCleanerThreads=1, defaultCleanupDelay(s)=300, routeBasedLogging=true
2017-05-01T13:04:23,135 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: TaskExecutorService is being setup with parameters: numExecutors=1, waitQueueSize=10, waitQueueComparatorClassName=org.apache.hadoop.hive.llap.daemon.impl.comparator.ShortestJobFirstComparator, enablePreemption=true
2017-05-01T13:04:23,159 INFO  [main ()] org.apache.tez.hadoop.shim.HadoopShimsLoader: Trying to locate HadoopShimProvider for hadoopVersion=2.7.3.2.6.0.3-8, majorVersion=2, minorVersion=7
2017-05-01T13:04:23,162 INFO  [main ()] org.apache.tez.hadoop.shim.HadoopShimsLoader: Picked HadoopShim org.apache.tez.hadoop.shim.HadoopShimsomehost, providerName=org.apache.tez.hadoop.shim.HadoopShimsomehostProvider, overrideProviderViaConfig=null, hadoopVersion=2.7.3.2.6.0.3-8, majorVersion=2, minorVersion=7
2017-05-01T13:04:23,162 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl: ContainerRunnerImpl config: memoryPerExecutorDerviced=796865344
2017-05-01T13:04:23,165 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Adding shutdown hook for LlapDaemon
2017-05-01T13:04:23,390 WARN  [main ()] org.apache.hadoop.hive.conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value
2017-05-01T13:04:23,514 INFO  [main ()] LlapIoImpl: Initializing LLAP IO in cache mode
2017-05-01T13:04:23,521 INFO  [main ()] org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics: Created interval PercentileDecodingTime_30s
2017-05-01T13:04:23,523 INFO  [main ()] LlapIoImpl: Started llap daemon metrics with displayName: LlapDaemonIOMetrics-somehost5.somedomain sessionId: 7b7897de-b8a8-48d5-b4a8-1143f5ef91ea
2017-05-01T13:04:23,525 INFO  [main ()] LlapIoImpl: LRFU cache policy with min buffer size 262144 and lambda 0.009999999776482582 (heap size 512)
2017-05-01T13:04:23,527 INFO  [main ()] LlapIoImpl: Memory manager initialized with max size 134217728 and  ability to evict blocks
2017-05-01T13:04:23,527 INFO  [main ()] LlapIoImpl: LRFU cache policy with min buffer size 262144 and lambda 0.009999999776482582 (heap size 321)
2017-05-01T13:04:23,527 INFO  [main ()] LlapIoImpl: Memory manager initialized with max size 83990936 and  ability to evict blocks
2017-05-01T13:04:23,531 INFO  [main ()] LlapIoImpl: Buddy allocator with direct buffers; allocation sizes 262144 - 16777216, arena size 16777216, total size 134217728
2017-05-01T13:04:23,551 INFO  [main ()] LlapIoImpl: Low level cache; cleanup interval 600 sec
2017-05-01T13:04:23,557 INFO  [main ()] org.apache.hadoop.service.AbstractService: Service LlapDaemon failed in state INITED; cause: java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl
java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) [hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
Caused by: java.lang.IllegalArgumentException
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
2017-05-01T13:04:23,562 WARN  [main ()] org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService: Stopping non-existent registry service
2017-05-01T13:04:23,563 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.AMReporter: Stopped service: org.apache.hadoop.hive.llap.daemon.impl.AMReporter
2017-05-01T13:04:23,563 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.QueryTracker: QueryTracker stopped
2017-05-01T13:04:23,563 INFO  [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: Wait-Queue-Scheduler-%d thread has been interrupted after shutdown.
2017-05-01T13:04:23,565 INFO  [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: Wait queue scheduler worker exited with success!
2017-05-01T13:04:23,569 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked
2017-05-01T13:04:23,569 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping LlapDaemon metrics system...
2017-05-01T13:04:23,570 INFO  [timeline ()] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: timeline thread interrupted.
2017-05-01T13:04:23,572 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system stopped.
2017-05-01T13:04:23,572 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system shutdown complete.
2017-05-01T13:04:23,577 WARN  [main ()] org.apache.hadoop.service.AbstractService: When stopping the service LlapDaemon : java.lang.IllegalStateException: LlapOutputFormatService must be started before invoking get
java.lang.IllegalStateException: LlapOutputFormatService must be started before invoking get
	at com.google.common.base.Preconditions.checkState(Preconditions.java:149) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.LlapOutputFormatService.get(LlapOutputFormatService.java:97) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceStop(LlapDaemon.java:437) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) [hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:171) [hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
2017-05-01T13:04:23,578 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception
java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
Caused by: java.lang.IllegalArgumentException
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
2017-05-01T13:04:23,578 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked
2017-05-01T13:04:23,579 WARN  [main ()] org.apache.hadoop.metrics2.util.MBeans: Error unregistering Hadoop:service=LlapDaemon,name=LlapDaemonInfo
javax.management.InstanceNotFoundException: Hadoop:service=LlapDaemon,name=LlapDaemonInfo
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) ~[?:1.8.0_112]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427) ~[?:1.8.0_112]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415) ~[?:1.8.0_112]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) ~[?:1.8.0_112]
	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:110) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.shutdown(LlapDaemon.java:445) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:537) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]

avatar
Guru

Can you share the hiveserver2 interactive logs ?

avatar
Contributor

I've shared hiveserver2 interactive logs already

if you know where more logs are located please inform me I can't find more logs than provided above, thank you

locate interactive
/usr/libexec/git-core/git-add--interactive
/usr/libexec/git-core/git-rebase--interactive
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.pyo
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.pyo
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.pyo
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.pyo
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.pyo
/var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hiveserver2-interactive-site.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-env.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-site.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/tez-interactive-site.xml
/var/lib/ambari-agent/tmp/start_hiveserver2_interactive_script

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Contributor

Yes hive.llap.io.threadpool.size was invalid, I set it as number of executors as you said and now everything works fine.

Thank you very much

avatar
New Contributor

In my case, even I set the values of "hive.llap.io.threadpool.size" and "hive.llap.daemon.num.executors" same.
Still I got the error like as given below..
"2018-07-10 20:56:26,421 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live Instances : '2'. Desired Instances : '3' after 619.594609976 secs."

When I changed "Number of Node(s) for running Hive LLAP daemon (num_llap_nodes_for_llap_daemons) " to 2.
It's working well.