Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

HiveServer2 Interactive Start failed

Labels:
avatar
author-rank shota
Contributor

Created ‎04-28-2017 12:29 PM

Hello I tried to activate LLAP and add HiveServer2 Interactive service but the service fails to start

I could not find any solution, any idea?

here is part of output in ambari when process is starting: 

2017-04-28 15:45:34,633 [main] INFO  util.ExitUtil - Exiting with status 0
2017-04-28 15:45:35,491 - Submitted LLAP app name : llap0
2017-04-28 15:45:35,492 - 





2017-04-28 15:45:35,492 - LLAP status command : /usr/hdp/current/hive-server2-hive2/bin/hive --service llapstatus -w -r 0.8 -i 2 -t 400
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARN conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value

LLAPSTATUS WatchMode with timeout=400 s
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances
--------------------------------------------------------------------------------
WARN cli.LlapStatusServiceDriver: Application stopped while launching. COMPLETE state reached while waiting for RUNNING state. Failing fast..
LLAP Application already complete. ApplicationId=application_1493310509760_0013
	FAILED container:  container_e23_1493310509760_0013_01_000003, Logs at: http://somehost1:19888/jobhistory/logs/somehostN1:45454/container_e23_1493310509760_0013_01_000003/c...
	FAILED container:  container_e23_1493310509760_0013_01_000002, Logs at: http://somehost1:19888/jobhistory/logs/somehostN2:45454/container_e23_1493310509760_0013_01_000002/c...
	FAILED container:  container_e23_1493310509760_0013_01_000005, Logs at: http://somehost1:19888/jobhistory/logs/somehostN3:45454/container_e23_1493310509760_0013_01_000005/c...
	FAILED container:  container_e23_1493310509760_0013_01_000004, Logs at: http://somehost1:19888/jobhistory/logs/somehostN4:45454/container_e23_1493310509760_0013_01_000004/c...
	FAILED container:  container_e23_1493310509760_0013_01_000007, Logs at: http://somehost1:19888/jobhistory/logs/somehostN5:45454/container_e23_1493310509760_0013_01_000007/c...
	FAILED container:  container_e23_1493310509760_0013_01_000006, Logs at: http://somehost1:19888/jobhistory/logs/somehostN6:45454/container_e23_1493310509760_0013_01_000006/c...
	FAILED container:  container_e23_1493310509760_0013_01_000009, Logs at: http://somehost1:19888/jobhistory/logs/---z---:45454/container_e23_1493310509760_0013_01_000009/ctx/...
	FAILED container:  container_e23_1493310509760_0013_01_000008, Logs at: http://somehost1:19888/jobhistory/logs/----z---:45454/container_e23_1493310509760_0013_01_000008/ctx...
	FAILED container:  container_e23_1493310509760_0013_01_000010, Logs at: http://somehost1:19888/jobhistory/logs/-----z----:45454/container_e23_1493310509760_0013_01_000010/c...
Unstable Application Instance : - failed with component LLAP failed 'recently' 6 times (6 in startup); threshold is 5 - last failure: Failure container_e23_1493310509760_0013_01_000007 on host somehostN12 (0): http://somehost1:19888/jobhistory/logs/somehostN12:45454/container_e23_1493310509760_0013_01_000007/...
--------------------------------------------------------------------------------
P.S. not real hostnames, I've deleted or changed
9,131 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank sergey
Rising Star

Created ‎05-30-2017 11:26 PM

Looks like hive.llap.io.threadpool.size is set to an invalid value. Are you using Ambari? Sometimes, depending on the cluster machine config, it may happen. You may need to set it manually in the advanced LLAP config in Hive configuration. The recommended default value is the same as number of executors. Is it also set to 0? If so, that would also need to be adjusted (typically to # of CPUs or based on memory, to allow 3-4-8Gb per executor). What version of Ambari is this?

View solution in original post

6,278 Views
8 REPLIES 8

avatar
author-rank deepesh1
Guru

Created ‎04-28-2017 02:13 PM

As the error message suggests, the containers launched by LLAP have failed, can you check the failed application logs?

6,278 Views

avatar
author-rank shota
Contributor

Created ‎05-01-2017 09:13 AM 

 Log Type: llap-daemon-hive-somehostname.log


Log Upload Time: Mon May 01 13:04:54 +0400 2017


Log Length: 16486


Showing 4096 bytes of 16486 total. Click here for the full log.


.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
2017-05-01T13:04:23,578 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception
java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
Caused by: java.lang.IllegalArgumentException
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
2017-05-01T13:04:23,578 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked
2017-05-01T13:04:23,579 WARN  [main ()] org.apache.hadoop.metrics2.util.MBeans: Error unregistering Hadoop:service=LlapDaemon,name=LlapDaemonInfo
javax.management.InstanceNotFoundException: Hadoop:service=LlapDaemon,name=LlapDaemonInfo
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) ~[?:1.8.0_112]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427) ~[?:1.8.0_112]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415) ~[?:1.8.0_112]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) ~[?:1.8.0_112]
	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:110) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.shutdown(LlapDaemon.java:445) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:537) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
6,278 Views
0 Kudos

avatar
author-rank shota
Contributor

Created ‎05-01-2017 09:18 AM

and this is full log of LLAP:

Log Type: llap-daemon-hive-somehost5.somedomain.log


Log Upload Time: Mon May 01 13:04:54 +0400 2017


Log Length: 16486


2017-05-01T13:04:21,632 INFO  [main ()] org.apache.hadoop.hive.conf.HiveConf: Found configuration file file:/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032/container_e23_1493310509760_0032_01_000007/app/install/conf/hive-site.xml
2017-05-01T13:04:22,044 INFO  [main ()] org.apache.hadoop.hive.llap.LlapUtil: Using local dirs from environment: /DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032
2017-05-01T13:04:22,200 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LLAP daemon logging initialized from file:/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032/container_e23_1493310509760_0032_01_000007/app/install/conf/llap-daemon-log4j2.properties in 153 ms. Async: true
2017-05-01T13:04:22,202 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: 


$$\       $$\        $$$$$$\  $$$$$$$\
$$ |      $$ |      $$  __$$\ $$  __$$\
$$ |      $$ |      $$ /  $$ |$$ |  $$ |
$$ |      $$ |      $$$$$$$$ |$$$$$$$  |
$$ |      $$ |      $$  __$$ |$$  ____/
$$ |      $$ |      $$ |  $$ |$$ |
$$$$$$$$\ $$$$$$$$\ $$ |  $$ |$$ |
\________|\________|\__|  \__|\__|




2017-05-01T13:04:22,203 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Zookeeper Quorum: somehost1.somedomain:2181,somehost2.somedomain:2181,somehost3.somedomain:2181
2017-05-01T13:04:22,462 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Starting daemon as user: hive
2017-05-01T13:04:22,475 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Attempting to start LlapDaemonConf with the following configuration: maxJvmMemory=839909370 (801.00MB), requestedExecutorMemory=838860800 (800.00MB), llapIoCacheSize=134217728 (128.00MB), xmxHeadRoomMemory=41995468 (40.05MB), adjustedExecutorMemory=796865332 (759.95MB), numExecutors=1, llapIoEnabled=true, llapIoCacheIsDirect=true, rpcListenerPort=0, mngListenerPort=15004, webPort=15002, outputFormatSvcPort=15003, workDirs=[/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032], shufflePort=15551, waitQueueSize= 10, enablePreemption= true
2017-05-01T13:04:22,546 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2-llapdaemon.properties
2017-05-01T13:04:22,800 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink.
2017-05-01T13:04:22,805 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Identified hostname = somehost5.somedomain, serviceName = llapdaemon
2017-05-01T13:04:22,870 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.availability.MetricSinkWriteShardHostnameHashingStrategy: Calculated collector shard somehost3.somedomain based on hostname: somehost5.somedomain
2017-05-01T13:04:22,871 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Collector Uri: http://somehost3.somedomain:6188/ws/v1/timeline/metrics
2017-05-01T13:04:22,871 INFO  [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Container Metrics Uri: http://somehost3.somedomain:6188/ws/v1/timeline/containermetrics
2017-05-01T13:04:22,884 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink timeline started
2017-05-01T13:04:22,903 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2017-05-01T13:04:22,904 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system started
2017-05-01T13:04:22,909 INFO  [org.apache.hadoop.util.JvmPauseMonitor$Monitor@324a0017 ()] org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
2017-05-01T13:04:22,940 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Started LlapMetricsSystem with displayName: LlapDaemonExecutorMetrics-somehost5.somedomain sessionId: 7b7897de-b8a8-48d5-b4a8-1143f5ef91ea
2017-05-01T13:04:22,954 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.AMReporter: Setting up AMReporter with heartbeatInterval(ms)=10000, retryTime(ms)=10000, retrySleep(ms)=2000
2017-05-01T13:04:22,966 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl: Creating: LlapProtocolServerImpl with port configured to: 0
2017-05-01T13:04:23,131 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.QueryTracker: QueryTracker setup with numCleanerThreads=1, defaultCleanupDelay(s)=300, routeBasedLogging=true
2017-05-01T13:04:23,135 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: TaskExecutorService is being setup with parameters: numExecutors=1, waitQueueSize=10, waitQueueComparatorClassName=org.apache.hadoop.hive.llap.daemon.impl.comparator.ShortestJobFirstComparator, enablePreemption=true
2017-05-01T13:04:23,159 INFO  [main ()] org.apache.tez.hadoop.shim.HadoopShimsLoader: Trying to locate HadoopShimProvider for hadoopVersion=2.7.3.2.6.0.3-8, majorVersion=2, minorVersion=7
2017-05-01T13:04:23,162 INFO  [main ()] org.apache.tez.hadoop.shim.HadoopShimsLoader: Picked HadoopShim org.apache.tez.hadoop.shim.HadoopShimsomehost, providerName=org.apache.tez.hadoop.shim.HadoopShimsomehostProvider, overrideProviderViaConfig=null, hadoopVersion=2.7.3.2.6.0.3-8, majorVersion=2, minorVersion=7
2017-05-01T13:04:23,162 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl: ContainerRunnerImpl config: memoryPerExecutorDerviced=796865344
2017-05-01T13:04:23,165 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Adding shutdown hook for LlapDaemon
2017-05-01T13:04:23,390 WARN  [main ()] org.apache.hadoop.hive.conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value
2017-05-01T13:04:23,514 INFO  [main ()] LlapIoImpl: Initializing LLAP IO in cache mode
2017-05-01T13:04:23,521 INFO  [main ()] org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics: Created interval PercentileDecodingTime_30s
2017-05-01T13:04:23,523 INFO  [main ()] LlapIoImpl: Started llap daemon metrics with displayName: LlapDaemonIOMetrics-somehost5.somedomain sessionId: 7b7897de-b8a8-48d5-b4a8-1143f5ef91ea
2017-05-01T13:04:23,525 INFO  [main ()] LlapIoImpl: LRFU cache policy with min buffer size 262144 and lambda 0.009999999776482582 (heap size 512)
2017-05-01T13:04:23,527 INFO  [main ()] LlapIoImpl: Memory manager initialized with max size 134217728 and  ability to evict blocks
2017-05-01T13:04:23,527 INFO  [main ()] LlapIoImpl: LRFU cache policy with min buffer size 262144 and lambda 0.009999999776482582 (heap size 321)
2017-05-01T13:04:23,527 INFO  [main ()] LlapIoImpl: Memory manager initialized with max size 83990936 and  ability to evict blocks
2017-05-01T13:04:23,531 INFO  [main ()] LlapIoImpl: Buddy allocator with direct buffers; allocation sizes 262144 - 16777216, arena size 16777216, total size 134217728
2017-05-01T13:04:23,551 INFO  [main ()] LlapIoImpl: Low level cache; cleanup interval 600 sec
2017-05-01T13:04:23,557 INFO  [main ()] org.apache.hadoop.service.AbstractService: Service LlapDaemon failed in state INITED; cause: java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl
java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) [hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
Caused by: java.lang.IllegalArgumentException
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
2017-05-01T13:04:23,562 WARN  [main ()] org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService: Stopping non-existent registry service
2017-05-01T13:04:23,563 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.AMReporter: Stopped service: org.apache.hadoop.hive.llap.daemon.impl.AMReporter
2017-05-01T13:04:23,563 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.QueryTracker: QueryTracker stopped
2017-05-01T13:04:23,563 INFO  [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: Wait-Queue-Scheduler-%d thread has been interrupted after shutdown.
2017-05-01T13:04:23,565 INFO  [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: Wait queue scheduler worker exited with success!
2017-05-01T13:04:23,569 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked
2017-05-01T13:04:23,569 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping LlapDaemon metrics system...
2017-05-01T13:04:23,570 INFO  [timeline ()] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: timeline thread interrupted.
2017-05-01T13:04:23,572 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system stopped.
2017-05-01T13:04:23,572 INFO  [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system shutdown complete.
2017-05-01T13:04:23,577 WARN  [main ()] org.apache.hadoop.service.AbstractService: When stopping the service LlapDaemon : java.lang.IllegalStateException: LlapOutputFormatService must be started before invoking get
java.lang.IllegalStateException: LlapOutputFormatService must be started before invoking get
	at com.google.common.base.Preconditions.checkState(Preconditions.java:149) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.LlapOutputFormatService.get(LlapOutputFormatService.java:97) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceStop(LlapDaemon.java:437) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) [hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:171) [hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
2017-05-01T13:04:23,578 WARN  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception
java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
Caused by: java.lang.IllegalArgumentException
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112]
	at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	... 4 more
2017-05-01T13:04:23,578 INFO  [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked
2017-05-01T13:04:23,579 WARN  [main ()] org.apache.hadoop.metrics2.util.MBeans: Error unregistering Hadoop:service=LlapDaemon,name=LlapDaemonInfo
javax.management.InstanceNotFoundException: Hadoop:service=LlapDaemon,name=LlapDaemonInfo
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) ~[?:1.8.0_112]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427) ~[?:1.8.0_112]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415) ~[?:1.8.0_112]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) ~[?:1.8.0_112]
	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:110) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.shutdown(LlapDaemon.java:445) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
	at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:537) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
6,278 Views
0 Kudos

avatar
author-rank srai1
Guru

Created ‎05-01-2017 06:42 PM

Can you share the hiveserver2 interactive logs ?

6,278 Views
0 Kudos

avatar
author-rank shota
Contributor

Created ‎05-02-2017 07:35 AM

I've shared hiveserver2 interactive logs already

if you know where more logs are located please inform me I can't find more logs than provided above, thank you

locate interactive
/usr/libexec/git-core/git-add--interactive
/usr/libexec/git-core/git-rebase--interactive
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.pyo
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.pyo
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.pyo
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.pyo
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.py
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.pyc
/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.pyo
/var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hiveserver2-interactive-site.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-env.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-site.xml
/var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/tez-interactive-site.xml
/var/lib/ambari-agent/tmp/start_hiveserver2_interactive_script
6,278 Views
0 Kudos

avatar
author-rank sergey
Rising Star

Created ‎05-30-2017 11:26 PM

Looks like hive.llap.io.threadpool.size is set to an invalid value. Are you using Ambari? Sometimes, depending on the cluster machine config, it may happen. You may need to set it manually in the advanced LLAP config in Hive configuration. The recommended default value is the same as number of executors. Is it also set to 0? If so, that would also need to be adjusted (typically to # of CPUs or based on memory, to allow 3-4-8Gb per executor). What version of Ambari is this?

6,279 Views

avatar
author-rank shota
Contributor

Created ‎09-05-2017 02:38 PM

Yes hive.llap.io.threadpool.size was invalid, I set it as number of executors as you said and now everything works fine.

Thank you very much

6,278 Views
0 Kudos

avatar
author-rank yeskko
New Contributor

Created ‎07-11-2018 01:44 AM

In my case, even I set the values of "hive.llap.io.threadpool.size" and "hive.llap.daemon.num.executors" same.
Still I got the error like as given below..
"2018-07-10 20:56:26,421 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live Instances : '2'. Desired Instances : '3' after 619.594609976 secs."

When I changed "Number of Node(s) for running Hive LLAP daemon (num_llap_nodes_for_llap_daemons) " to 2.
It's working well.

6,278 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements