Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive2 Interactive LLAP fails to start

Highlighted

Hive2 Interactive LLAP fails to start

New Contributor

I have been trying to start Hive2 Interactive on HDP 2.6.5 that ships with Hive 2.1.0. I have followed the guidelines mentioned in the following articles about how to size and tune LLAP with Yarn

https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html

But LLAP daemon fails to start and Hive interactive gives up after configured retries. Also followed the following steps to troubleshoot and the error doesn't seem to be sizing related.

https://community.hortonworks.com/articles/149899/investigating-when-llap-doesnt-start.html

Following is my cluster setup in brief:
hive.llap.daemon.yarn.container.mb= 10 GB
llap_heap_size= 7 GB
In-Memory Cache per Daemon= 2 GB
Head room= 1 GB (hive.llap.daemon.yarn.container.mb should be = llap_heap_size + In-Memory Cache per Daemon + Headroom)
Yarn container max= 14.6 GB, min 1 GB
Number of executors per LLAP Daemon= 1
Yarn pre-emption enabled
Retries for checking LLAP status= 20
Following are the startup script logs and Yarn container relevant logs.
https://pastebin.com/raw/4ywJHp4M
https://pastebin.com/raw/AiJkfwUR
Following is a snapshot of Slide AM page

Application: llap0

Statusall containers allocated
Total number of containers
Create time:17 Jan 2019 05:40:15 GMT
Running since:17 Jan 2019 05:40:15 GMT
Time last flexed:N/A
Application storage path:hdfs://hsft/user/hive/.slider/cluster/llap0/database
Application configuration path: hdfs://hsft/user/hive/.slider/cluster/llap0/snapshot

Component Instances

ComponentDesired Actual Outstanding Requests Failed Failed to start Placement
LLAP
slider-appmaster

Application Container Diagnostics

Container IDComponent State Exit Code Logs Diagnostics
container_e27_1547703090681_0001_01_000002LLAP3-1000Logs
container_e27_1547703090681_0001_01_000003LLAP3-1000Logs
container_e27_1547703090681_0001_01_000004LLAP3-1000Logs
container_e27_1547703090681_0001_01_000005LLAP3-1000Logs
2 REPLIES 2

Re: Hive2 Interactive LLAP fails to start

@Abhishek Gupta I'm seeing a permission denied error in the logs.

/hadoop/yarn/local/usercache/hive/appcache/application_1547703090681_0001/container_e27_1547703090681_0001_01_000004/app/install//bin/runLlapDaemon.sh: Permission denied

Re: Hive2 Interactive LLAP fails to start

New Contributor

Yes, I did see this error when ambari issues a nohup nice on this script. However I am not sure why is this happening.

Another error that is striking is the error by NetUtil.py failing to connect with Slide agent service as the OS Python version 2.7.6 where module ssl doesn’t have the attribute `_create_unverified_context`, this is resulting in the following error in slideragent log

INFO 2019-01-17 09:04:59,889 NetUtil.py:66 - Failed to connect to https://shdp-ycn04:38097/ws/v1/slider/agents/ due to 'module' object has no attribute '_create_unverified_context' 
INFO 2019-01-17 09:04:59,889 NetUtil.py:85 - Server at https://shdp-ycn04:38097/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...