Hive server2 interactive(HSI Server) failed to start in HDP3.0 after enabling the Hive LLAP:

[#1]

Please verify the below Logs:

hive.llap.daemon.yarn.container.mb hive.llap.daemon.num.executors hive.llap.io.memory.size re=[/usr/jdk64/jdk1.8.0_112/jre] 2018-08-13T06:01:46,947 WARN [llap-pkg-3] conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist 2018-08-13T06:01:46,948 WARN [llap-pkg-3] conf.HiveConf: HiveConf of name hive.heapsize does not exist 2018-08-13T06:01:46,949 WARN [llap-pkg-3] conf.HiveConf: HiveConf of name hive.druid.select.distribute does not exist 2018-08-13T06:01:49,040 WARN [llap-pkg-1] conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist 2018-08-13T06:01:49,041 WARN [llap-pkg-1] conf.HiveConf: HiveConf of name hive.heapsize does not exist 2018-08-13T06:01:49,041 WARN [llap-pkg-1] conf.HiveConf: HiveConf of name hive.druid.select.distribute does not exist 2018-08-13T06:02:20,524 WARN [main] curator.CuratorZookeeperClient: session timeout [10000] is less than connection timeout [15000] 2018-08-13T06:02:30,833 WARN [main] conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist 2018-08-13T06:02:30,839 WARN [main] conf.HiveConf: HiveConf of name hive.heapsize does not exist 2018-08-13T06:02:30,840 WARN [main] conf.HiveConf: HiveConf of name hive.druid.select.distribute does not exist 2018-08-13T06:02:33,327 INFO [main] LlapStatusServiceDriverConsole: LLAPSTATUS WatchMode with timeout=400 s 2018-08-13T06:02:33,328 INFO [main] LlapStatusServiceDriverConsole: -------------------------------------------------------------------------------- 2018-08-13T06:02:33,328 INFO [main] LlapStatusServiceDriverConsole: LLAP status unknown 2018-08-13T06:02:33,329 INFO [main] LlapStatusServiceDriverConsole: -------------------------------------------------------------------------------- 2018-08-13T06:02:33,329 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2018-08-13T06:02:35,337 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2018-08-13T06:02:37,343 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2018-08-13T06:02:39,350 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2018-08-13T06:02:41,358 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2018-08-13T06:02:43,365 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2018-08-13T06:02:45,375 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2018-08-13T06:02:47,382 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2018-08-13T06:02:49,448 INFO [main] LlapStatusServiceDriverConsole: LLAP status unknown 2018-08-13T06:02:49,448 INFO [main] LlapStatusServiceDriverConsole: -------------------------------------------------------------------------------- 2018-08-13T06:02:49,449 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying..

[#2 · Accepted solution]

@subhash parise There are quite a few errors above of missing settings. I would confirm that you have completed the Hive LLAP setup and try to restart all again. Some slight modifications may be necessary to fit your specific cluster, but the following links may be helpful:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_command-line-installation/content/instal...

https://community.hortonworks.com/articles/114636/how-to-enable-hive-llap-for-interactive-sql-on-had...

Hive LLAP can be very tricky so be sure to follow all of the steps and test restarting LLAP a few times.

If this answer is helpful please choose ACCEPT.

[#3]

Hi @Steven Matison,

Appreciate your help. i have tried recommended steps but still no lucl

below are my single node cluster configuration details:

hive.tez.container.size -29 G.B

LLAP Queue Capacity(50%)

yarn.nodemanager.resource.memory-mb -174 G.B

yar.scheduler.maximum-allocation-mb - 168 G.B

yarn.scheduler.minimum-allocation-mb - 1 G.B

hive.server2.tez.sessions.per.default.queue = 1

tez.am.resource.memory.mb = 4096

hive.llap.daemon.num.executors=2

hive.llap.daemon.yarn.container.mb = 168 g.b

In- memory cache per daemon - 143 g.B

number of executor per llap daemon = 2

llap daemon heap size- 24064

llap daemon max container head room = 12288

Please suggest me if i have got missed any other configuration or miss configuration here ?

Thank you in advance.

[#4]

@subhash parise

click on Comment to reply In-Line, versus making a new answer...

Some of your settings which above over 100 GB are concerning. I would recommend starting with much smaller settings: 2gb, 4gb, 8gb, 16gb, 32gb etc. Find a working combination then experiment with increasing attributes one at time especially with over 100 gb settings.

My settings are:

hive.tez.container.size: 13472 MB

Number of nodes used by Hive's LLAP: 2

Memory Per Daemon: 43520

In-Memory cache Per daemon: 2560

Number of executors per LLAP Daemon: 10

Hive Tez Container Size: 4096

Number containers held: 1

Hiver Server Interactive Heap Size: 2048 MB

LLAP Daemon Container max Headroom: 12288 MB

LLAP Daemon Heap Size: 32768 MB

Slider AM container size: 2560 MB

With 100s of GB of ram available in your cluster, you should be able to get to 10 Nodes used by LLAP (10 with smaller settings - versus 1 with huge settings), but start at 2, get them working, and build up from there.

[#5]

@subhash parise

i having same issues, do you solve it?

may i know what your solution for this.

[#6]

@yong lau

I seen your other post, but the error was hard to find in the bulk paste. Try posting your error in a

code box

so that we can see it better. Also be sure to dial into the actual error versus sending needless text.

That said, HIVE LLAP requires some configurations we outline above in this post. Check those out as well as the links above Make sure you have the settings for low specs and for a single LLAP container and try to start LLAP. Sometimes it takes me 2-3 times to start without errors. Once you have it working with low specs, slowly increase specs. It is also important to know that the actual errors you need to find, are likely inside of the YARN containers, so you will have to dig them out to truely know the issue that actually stops it from starting.