Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
Go to solution
HiveServer2 Interactive Start failed
Labels:
- Labels:
-
Apache Hive
Contributor
Created ‎04-28-2017 12:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello I tried to activate LLAP and add HiveServer2 Interactive service but the service fails to start
I could not find any solution, any idea?
here is part of output in ambari when process is starting:
2017-04-28 15:45:34,633 [main] INFO util.ExitUtil - Exiting with status 0 2017-04-28 15:45:35,491 - Submitted LLAP app name : llap0 2017-04-28 15:45:35,492 - 2017-04-28 15:45:35,492 - LLAP status command : /usr/hdp/current/hive-server2-hive2/bin/hive --service llapstatus -w -r 0.8 -i 2 -t 400 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] WARN conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value LLAPSTATUS WatchMode with timeout=400 s -------------------------------------------------------------------------------- LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances -------------------------------------------------------------------------------- LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances -------------------------------------------------------------------------------- LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances -------------------------------------------------------------------------------- LLAP Starting up with AppId=application_1493310509760_0013. Started 0/6 instances -------------------------------------------------------------------------------- WARN cli.LlapStatusServiceDriver: Application stopped while launching. COMPLETE state reached while waiting for RUNNING state. Failing fast.. LLAP Application already complete. ApplicationId=application_1493310509760_0013 FAILED container: container_e23_1493310509760_0013_01_000003, Logs at: http://somehost1:19888/jobhistory/logs/somehostN1:45454/container_e23_1493310509760_0013_01_000003/c... FAILED container: container_e23_1493310509760_0013_01_000002, Logs at: http://somehost1:19888/jobhistory/logs/somehostN2:45454/container_e23_1493310509760_0013_01_000002/c... FAILED container: container_e23_1493310509760_0013_01_000005, Logs at: http://somehost1:19888/jobhistory/logs/somehostN3:45454/container_e23_1493310509760_0013_01_000005/c... FAILED container: container_e23_1493310509760_0013_01_000004, Logs at: http://somehost1:19888/jobhistory/logs/somehostN4:45454/container_e23_1493310509760_0013_01_000004/c... FAILED container: container_e23_1493310509760_0013_01_000007, Logs at: http://somehost1:19888/jobhistory/logs/somehostN5:45454/container_e23_1493310509760_0013_01_000007/c... FAILED container: container_e23_1493310509760_0013_01_000006, Logs at: http://somehost1:19888/jobhistory/logs/somehostN6:45454/container_e23_1493310509760_0013_01_000006/c... FAILED container: container_e23_1493310509760_0013_01_000009, Logs at: http://somehost1:19888/jobhistory/logs/---z---:45454/container_e23_1493310509760_0013_01_000009/ctx/... FAILED container: container_e23_1493310509760_0013_01_000008, Logs at: http://somehost1:19888/jobhistory/logs/----z---:45454/container_e23_1493310509760_0013_01_000008/ctx... FAILED container: container_e23_1493310509760_0013_01_000010, Logs at: http://somehost1:19888/jobhistory/logs/-----z----:45454/container_e23_1493310509760_0013_01_000010/c... Unstable Application Instance : - failed with component LLAP failed 'recently' 6 times (6 in startup); threshold is 5 - last failure: Failure container_e23_1493310509760_0013_01_000007 on host somehostN12 (0): http://somehost1:19888/jobhistory/logs/somehostN12:45454/container_e23_1493310509760_0013_01_000007/... -------------------------------------------------------------------------------- P.S. not real hostnames, I've deleted or changed
1 ACCEPTED SOLUTION
Rising Star
Created ‎05-30-2017 11:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Looks like hive.llap.io.threadpool.size is set to an invalid value. Are you using Ambari? Sometimes, depending on the cluster machine config, it may happen. You may need to set it manually in the advanced LLAP config in Hive configuration. The recommended default value is the same as number of executors. Is it also set to 0? If so, that would also need to be adjusted (typically to # of CPUs or based on memory, to allow 3-4-8Gb per executor). What version of Ambari is this?
8 REPLIES 8
Guru
Created ‎04-28-2017 02:13 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As the error message suggests, the containers launched by LLAP have failed, can you check the failed application logs?
Contributor
Created ‎05-01-2017 09:13 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Log Type: llap-daemon-hive-somehostname.log Log Upload Time: Mon May 01 13:04:54 +0400 2017 Log Length: 16486 Showing 4096 bytes of 16486 total. Click here for the full log. .llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] 2017-05-01T13:04:23,578 WARN [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 4 more Caused by: java.lang.IllegalArgumentException at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 4 more 2017-05-01T13:04:23,578 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked 2017-05-01T13:04:23,579 WARN [main ()] org.apache.hadoop.metrics2.util.MBeans: Error unregistering Hadoop:service=LlapDaemon,name=LlapDaemonInfo javax.management.InstanceNotFoundException: Hadoop:service=LlapDaemon,name=LlapDaemonInfo at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) ~[?:1.8.0_112] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427) ~[?:1.8.0_112] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415) ~[?:1.8.0_112] at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) ~[?:1.8.0_112] at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:110) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.shutdown(LlapDaemon.java:445) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:537) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
Contributor
Created ‎05-01-2017 09:18 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
and this is full log of LLAP:
Log Type: llap-daemon-hive-somehost5.somedomain.log Log Upload Time: Mon May 01 13:04:54 +0400 2017 Log Length: 16486 2017-05-01T13:04:21,632 INFO [main ()] org.apache.hadoop.hive.conf.HiveConf: Found configuration file file:/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032/container_e23_1493310509760_0032_01_000007/app/install/conf/hive-site.xml 2017-05-01T13:04:22,044 INFO [main ()] org.apache.hadoop.hive.llap.LlapUtil: Using local dirs from environment: /DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032 2017-05-01T13:04:22,200 WARN [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LLAP daemon logging initialized from file:/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032/container_e23_1493310509760_0032_01_000007/app/install/conf/llap-daemon-log4j2.properties in 153 ms. Async: true 2017-05-01T13:04:22,202 WARN [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: $$\ $$\ $$$$$$\ $$$$$$$\ $$ | $$ | $$ __$$\ $$ __$$\ $$ | $$ | $$ / $$ |$$ | $$ | $$ | $$ | $$$$$$$$ |$$$$$$$ | $$ | $$ | $$ __$$ |$$ ____/ $$ | $$ | $$ | $$ |$$ | $$$$$$$$\ $$$$$$$$\ $$ | $$ |$$ | \________|\________|\__| \__|\__| 2017-05-01T13:04:22,203 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Zookeeper Quorum: somehost1.somedomain:2181,somehost2.somedomain:2181,somehost3.somedomain:2181 2017-05-01T13:04:22,462 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Starting daemon as user: hive 2017-05-01T13:04:22,475 WARN [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Attempting to start LlapDaemonConf with the following configuration: maxJvmMemory=839909370 (801.00MB), requestedExecutorMemory=838860800 (800.00MB), llapIoCacheSize=134217728 (128.00MB), xmxHeadRoomMemory=41995468 (40.05MB), adjustedExecutorMemory=796865332 (759.95MB), numExecutors=1, llapIoEnabled=true, llapIoCacheIsDirect=true, rpcListenerPort=0, mngListenerPort=15004, webPort=15002, outputFormatSvcPort=15003, workDirs=[/DATA/hadoop/yarn/local/usercache/hive/appcache/application_1493310509760_0032], shufflePort=15551, waitQueueSize= 10, enablePreemption= true 2017-05-01T13:04:22,546 INFO [main ()] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2-llapdaemon.properties 2017-05-01T13:04:22,800 INFO [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2017-05-01T13:04:22,805 INFO [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Identified hostname = somehost5.somedomain, serviceName = llapdaemon 2017-05-01T13:04:22,870 INFO [main ()] org.apache.hadoop.metrics2.sink.timeline.availability.MetricSinkWriteShardHostnameHashingStrategy: Calculated collector shard somehost3.somedomain based on hostname: somehost5.somedomain 2017-05-01T13:04:22,871 INFO [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Collector Uri: http://somehost3.somedomain:6188/ws/v1/timeline/metrics 2017-05-01T13:04:22,871 INFO [main ()] org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink: Container Metrics Uri: http://somehost3.somedomain:6188/ws/v1/timeline/containermetrics 2017-05-01T13:04:22,884 INFO [main ()] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink timeline started 2017-05-01T13:04:22,903 INFO [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-05-01T13:04:22,904 INFO [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system started 2017-05-01T13:04:22,909 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@324a0017 ()] org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor 2017-05-01T13:04:22,940 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Started LlapMetricsSystem with displayName: LlapDaemonExecutorMetrics-somehost5.somedomain sessionId: 7b7897de-b8a8-48d5-b4a8-1143f5ef91ea 2017-05-01T13:04:22,954 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.AMReporter: Setting up AMReporter with heartbeatInterval(ms)=10000, retryTime(ms)=10000, retrySleep(ms)=2000 2017-05-01T13:04:22,966 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl: Creating: LlapProtocolServerImpl with port configured to: 0 2017-05-01T13:04:23,131 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.QueryTracker: QueryTracker setup with numCleanerThreads=1, defaultCleanupDelay(s)=300, routeBasedLogging=true 2017-05-01T13:04:23,135 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: TaskExecutorService is being setup with parameters: numExecutors=1, waitQueueSize=10, waitQueueComparatorClassName=org.apache.hadoop.hive.llap.daemon.impl.comparator.ShortestJobFirstComparator, enablePreemption=true 2017-05-01T13:04:23,159 INFO [main ()] org.apache.tez.hadoop.shim.HadoopShimsLoader: Trying to locate HadoopShimProvider for hadoopVersion=2.7.3.2.6.0.3-8, majorVersion=2, minorVersion=7 2017-05-01T13:04:23,162 INFO [main ()] org.apache.tez.hadoop.shim.HadoopShimsLoader: Picked HadoopShim org.apache.tez.hadoop.shim.HadoopShimsomehost, providerName=org.apache.tez.hadoop.shim.HadoopShimsomehostProvider, overrideProviderViaConfig=null, hadoopVersion=2.7.3.2.6.0.3-8, majorVersion=2, minorVersion=7 2017-05-01T13:04:23,162 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl: ContainerRunnerImpl config: memoryPerExecutorDerviced=796865344 2017-05-01T13:04:23,165 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Adding shutdown hook for LlapDaemon 2017-05-01T13:04:23,390 WARN [main ()] org.apache.hadoop.hive.conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value 2017-05-01T13:04:23,514 INFO [main ()] LlapIoImpl: Initializing LLAP IO in cache mode 2017-05-01T13:04:23,521 INFO [main ()] org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics: Created interval PercentileDecodingTime_30s 2017-05-01T13:04:23,523 INFO [main ()] LlapIoImpl: Started llap daemon metrics with displayName: LlapDaemonIOMetrics-somehost5.somedomain sessionId: 7b7897de-b8a8-48d5-b4a8-1143f5ef91ea 2017-05-01T13:04:23,525 INFO [main ()] LlapIoImpl: LRFU cache policy with min buffer size 262144 and lambda 0.009999999776482582 (heap size 512) 2017-05-01T13:04:23,527 INFO [main ()] LlapIoImpl: Memory manager initialized with max size 134217728 and ability to evict blocks 2017-05-01T13:04:23,527 INFO [main ()] LlapIoImpl: LRFU cache policy with min buffer size 262144 and lambda 0.009999999776482582 (heap size 321) 2017-05-01T13:04:23,527 INFO [main ()] LlapIoImpl: Memory manager initialized with max size 83990936 and ability to evict blocks 2017-05-01T13:04:23,531 INFO [main ()] LlapIoImpl: Buddy allocator with direct buffers; allocation sizes 262144 - 16777216, arena size 16777216, total size 134217728 2017-05-01T13:04:23,551 INFO [main ()] LlapIoImpl: Low level cache; cleanup interval 600 sec 2017-05-01T13:04:23,557 INFO [main ()] org.apache.hadoop.service.AbstractService: Service LlapDaemon failed in state INITED; cause: java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) [hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 4 more Caused by: java.lang.IllegalArgumentException at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 4 more 2017-05-01T13:04:23,562 WARN [main ()] org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService: Stopping non-existent registry service 2017-05-01T13:04:23,563 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.AMReporter: Stopped service: org.apache.hadoop.hive.llap.daemon.impl.AMReporter 2017-05-01T13:04:23,563 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.QueryTracker: QueryTracker stopped 2017-05-01T13:04:23,563 INFO [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: Wait-Queue-Scheduler-%d thread has been interrupted after shutdown. 2017-05-01T13:04:23,565 INFO [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService: Wait queue scheduler worker exited with success! 2017-05-01T13:04:23,569 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked 2017-05-01T13:04:23,569 INFO [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping LlapDaemon metrics system... 2017-05-01T13:04:23,570 INFO [timeline ()] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: timeline thread interrupted. 2017-05-01T13:04:23,572 INFO [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system stopped. 2017-05-01T13:04:23,572 INFO [main ()] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: LlapDaemon metrics system shutdown complete. 2017-05-01T13:04:23,577 WARN [main ()] org.apache.hadoop.service.AbstractService: When stopping the service LlapDaemon : java.lang.IllegalStateException: LlapOutputFormatService must be started before invoking get java.lang.IllegalStateException: LlapOutputFormatService must be started before invoking get at com.google.common.base.Preconditions.checkState(Preconditions.java:149) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.LlapOutputFormatService.get(LlapOutputFormatService.java:97) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceStop(LlapDaemon.java:437) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) [hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:171) [hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] 2017-05-01T13:04:23,578 WARN [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception java.lang.RuntimeException: Failed to create org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:61) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.io.api.LlapProxy.initializeLlapIo(LlapProxy.java:50) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceInit(LlapDaemon.java:393) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 4 more Caused by: java.lang.IllegalArgumentException at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:67) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool.<init>(StatsRecordingThreadPool.java:59) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.<init>(LlapIoImpl.java:181) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_112] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_112] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_112] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_112] at org.apache.hadoop.hive.llap.io.api.LlapProxy.createInstance(LlapProxy.java:59) ~[hive-exec-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] ... 4 more 2017-05-01T13:04:23,578 INFO [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: LlapDaemon shutdown invoked 2017-05-01T13:04:23,579 WARN [main ()] org.apache.hadoop.metrics2.util.MBeans: Error unregistering Hadoop:service=LlapDaemon,name=LlapDaemonInfo javax.management.InstanceNotFoundException: Hadoop:service=LlapDaemon,name=LlapDaemonInfo at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) ~[?:1.8.0_112] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427) ~[?:1.8.0_112] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415) ~[?:1.8.0_112] at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) ~[?:1.8.0_112] at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:110) ~[hadoop-common-2.7.3.2.6.0.3-8.jar:?] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.shutdown(LlapDaemon.java:445) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:537) ~[hive-llap-server-2.1.0.2.6.0.3-8.jar:2.1.0.2.6.0.3-8]
Guru
Created ‎05-01-2017 06:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you share the hiveserver2 interactive logs ?
Contributor
Created ‎05-02-2017 07:35 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've shared hiveserver2 interactive logs already
if you know where more logs are located please inform me I can't find more logs than provided above, thank you
locate interactive /usr/libexec/git-core/git-add--interactive /usr/libexec/git-core/git-rebase--interactive /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.py /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.pyc /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_interactive_thrift_port.pyo /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.py /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.pyc /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.pyo /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.pyc /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.pyo /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.py /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.pyc /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.pyo /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.py /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.pyc /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.pyo /var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml /var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml /var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/hiveserver2-interactive-site.xml /var/lib/ambari-agent/cache/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml /var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-env.xml /var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-site.xml /var/lib/ambari-agent/cache/stacks/HDP/2.6/services/HIVE/configuration/tez-interactive-site.xml /var/lib/ambari-agent/tmp/start_hiveserver2_interactive_script
Rising Star
Created ‎05-30-2017 11:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Looks like hive.llap.io.threadpool.size is set to an invalid value. Are you using Ambari? Sometimes, depending on the cluster machine config, it may happen. You may need to set it manually in the advanced LLAP config in Hive configuration. The recommended default value is the same as number of executors. Is it also set to 0? If so, that would also need to be adjusted (typically to # of CPUs or based on memory, to allow 3-4-8Gb per executor). What version of Ambari is this?
Contributor
Created ‎09-05-2017 02:38 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes hive.llap.io.threadpool.size was invalid, I set it as number of executors as you said and now everything works fine.
Thank you very much
New Contributor
Created ‎07-11-2018 01:44 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In my case, even I set the values of "hive.llap.io.threadpool.size" and "hive.llap.daemon.num.executors" same.
Still I got the error like as given below..
"2018-07-10 20:56:26,421 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live Instances : '2'. Desired Instances : '3' after 619.594609976 secs."
When I changed "Number of Node(s) for running Hive LLAP daemon (num_llap_nodes_for_llap_daemons) " to 2.
It's working well.
