Support Questions
Find answers, ask questions, and share your expertise

LLAP: HiveServer2 Interactive Service is not starting

LLAP: HiveServer2 Interactive Service is not starting

HiveServer2 Interactive Service is not starting.

Error:

018-07-12 02:42:15,431 - LLAP status command : /usr/hdp/current/hive-server2-hive2/bin/hive --service llapstatus -w -r 0.8 -i 2 -t 400
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.3.0-235/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARN conf.HiveConf: HiveConf hive.llap.daemon.vcpus.per.instance expects INT type value

LLAPSTATUS WatchMode with timeout=400 s
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002.
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0002. Started 0/1 instances
--------------------------------------------------------------------------------
{
  "amInfo" : {
    "appName" : "llap0",
    "appType" : "org-apache-slider",
    "appId" : "application_1531338062331_0002",
    "containerId" : "container_e18_1531338062331_0002_01_000001",
    "hostname" : "hdp-3-dn1.com",
    "amWebUrl" : "http://hdp-3-dn1.com:59384/"
  },
  "state" : "LAUNCHING",
  "originalConfigurationPath" : "hdfs://hdp-1-nn.com:8020/user/hive/.slider/cluster/llap0/snapshot",
  "generatedConfigurationPath" : "hdfs://hdp-1-nn.com:8020/user/hive/.slider/cluster/llap0/generated",
  "desiredInstances" : 1,
  "liveInstances" : 0,
  "appStartTime" : 1531345352331,
  "runningThresholdAchieved" : false
}
WARN cli.LlapStatusServiceDriver: Watch timeout 400s exhausted before desired state RUNNING is attained.
2018-07-12 02:49:13,111 - LLAP app 'llap0' current state is LAUNCHING.
2018-07-12 02:49:13,111 - LLAP app 'llap0' current state is LAUNCHING.
2018-07-12 02:49:13,111 - LLAP app 'llap0' deployment unsuccessful.
2018-07-12 02:49:13,111 - Stopping LLAP
2018-07-12 02:49:13,112 - call[['slider', 'stop', 'llap0']] {'logoutput': True, 'user': 'hive', 'stderr': -1}
2018-07-12 02:49:22,903 [main] WARN  shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2018-07-12 02:49:22,945 [main] INFO  client.RMProxy - Connecting to ResourceManager at hdp-1-nn.com/192.168.100.10:8050
2018-07-12 02:49:23,545 [main] INFO  client.AHSProxy - Connecting to Application History server at hdp-1-nn.com/192.168.100.10:10200
2018-07-12 02:49:24,775 [main] INFO  util.ExitUtil - Exiting with status 0
2018-07-12 02:49:25,717 - call returned (0, '2018-07-12 02:49:22,903 [main] WARN  shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.\n2018-07-12 02:49:22,945 [main] INFO  client.RMProxy - Connecting to ResourceManager at hdp-1-nn.com/192.168.100.10:8050\n2018-07-12 02:49:23,545 [main] INFO  client.AHSProxy - Connecting to Application History server at hdp-1-nn.com/192.168.100.10:10200\n2018-07-12 02:49:24,775 [main] INFO  util.ExitUtil - Exiting with status 0', '')
2018-07-12 02:49:25,718 - Stopped llap0 application on Slider successfully
2018-07-12 02:49:25,718 - Execute[('slider', 'destroy', 'llap0', '--force')] {'ignore_failures': True, 'user': 'hive', 'timeout': 30}


5 REPLIES 5

Re: LLAP: HiveServer2 Interactive Service is not starting

Hortonworks Multinode Cluster

VMs Spec

  • VM#1: Active NameNode (32 GB RAM & 2 processors/ CPU)
  • VM#2: Standby NameNode (12 GB RAM & 1 processors/ CPU)
  • VM#3: DataNode (12 GB RAM & 1 processors/ CPU)

Other details:

  • OS: Linux 6.5
  • HDP 2.6.3 + Ambari 2.6.0.0
  • HDF 3.0.2 (only NiFi with min 3 GB and max 4 GB, No SSL)
  • Cluster with Kerberos (disabled)

---------------------------------------------------------------------------------------------------

For LLAP, did following things:

  • Pre-emption = Enabled
  • Capacity Schedule:
    • default: min 50% and max 100%
    • Added a new queue: llap with min 50% and max 50%
  • Memory allocated for all YARN containers on a node = 9 GB
  • Minimum Container Size (Memory) = 1 GB
  • Maximum Container Size (Memory) = 9 GB
  • Tez Container Size = 3 GB
  • HiveServer2 Heap Size = 2 GB
  • Metastore Heap Size= 2 GB
  • Client Heap Size = 1 GB
  • Enabled LLAP
    • Interactive Query Queue = llap
    • Number of nodes used by Hive's LLAP = 1
    • Maximum Total Concurrent Queries = 1
    • Memory per Daemon = 7168
    • In-Memory Cache per Daemon = 5120
    • Number of executors per LLAP Daemon = 1

  • Installed LLAP on Active NameNode as it took it as default
  • HiveServer2 Interactive = failed

------------------------------------

Further mode changed

  • tez.container.max.java.heap.fraction = 0.8 from -1

Not sure what i am missing. Looks like have done most of the things. Must be missing something special.

Looking forward for solution.

Cheers.....

Re: LLAP: HiveServer2 Interactive Service is not starting

@Mustafa Ali Qizilbash

Have you tried increasing the number of retries? It could just be that the default timeout of 400s is not enough for your environment. See also: https://community.hortonworks.com/content/supportkb/192159/how-to-increase-llap-application-status-c...

Advanced hive-interactive-env --> Number of retries while checking LLAP app status

Re: LLAP: HiveServer2 Interactive Service is not starting

Yes increased from 20 to 30 = 600s but still failing.

LLAPSTATUS WatchMode with timeout=600 s
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0005.
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0005. Started 0/1 instances
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1531338062331_0005. Started 0/1 instances

Re: LLAP: HiveServer2 Interactive Service is not starting

I keep getting this warning:

79452-llap-warning.jpeg

Does this has anything to do?

Re: LLAP: HiveServer2 Interactive Service is not starting

@Geoffrey Shelton Okot