Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive HiveServer2 Interactive LLAP doesn't start

Hive HiveServer2 Interactive LLAP doesn't start

New Contributor

Traceback (most recent call last):

  I use HDP 3.0, Hive LLAP doesn't start for interactive query
I tried many suggestions:
ref:https://community.hortonworks.com/questions/196210/hive-hiveserver2-interactive-llap-doesnt-start.html
Log my error: File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart self.status(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server_interactive.py", line 159, in status check_process_status(status_params.hive_interactive_pid) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning() ComponentIsNotRunning The above exception was the cause of the following exception: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server_interactive.py", line 541, in <module> HiveServerInteractive().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server_interactive.py", line 99, in start raise Fail("Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.") resource_management.core.exceptions.Fail: Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED. LLAP status unknown -------------------------------------------------------------------------------- WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. WARN cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. LLAP status unknown -------------------------------------------------------------------------------- { "amInfo" : { "appName" : "llap0", "appType" : "yarn-service", "appId" : "application_1535704246854_0006" }, "state" : "UNKNOWN", "desiredInstances" : 3, "liveInstances" : 3, "appStartTime" : 0, "runningThresholdAchieved" : false, "runningInstances" : [ { "hostname" : "ants-ctrl-el7-10-199-0-115", "containerId" : "container_e15_1535704246854_0006_01_000106", "yarnContainerExitStatus" : 0 }, { "hostname" : "ants-ctrl-el7-10-199-0-115", "containerId" : "container_e15_1535704246854_0006_01_000107", "yarnContainerExitStatus" : 0 }, { "hostname" : "ants-ctrl-el7-10-199-0-115", "containerId" : "container_e15_1535704246854_0006_01_000108", "yarnContainerExitStatus" : 0 } ] } WARN cli.LlapStatusServiceDriver: Watch timeout 400s exhausted before desired state RUNNING is attained. Command failed after 1 tries
4 REPLIES 4

Re: Hive HiveServer2 Interactive LLAP doesn't start

Hi @beenin ant ,

referring to this at log :

WARN cli.LlapStatusServiceDriver: Watch timeout 400s exhausted before desired state RUNNING is attained.

why i would suggest you to try increasing Ambari -> Hive -> Configs -> Advanced -> Advanced hive-interactive-env -> Number of retries while checking LLAP app status to double than its current value and see how if it works?

Please accept answer if this helped

Re: Hive HiveServer2 Interactive LLAP doesn't start

New Contributor

Thank you for your support.
I followed by you but it it not working.

WARN cli.LlapStatusServiceDriver: Watch timeout 600s exhausted before desired state RUNNING is attained.
Highlighted

Re: Hive HiveServer2 Interactive LLAP doesn't start

Super Collaborator

@beenin ant When troubleshooting LLAP startup issues it is important to fully investigate all of the logs. Without seeing the actual log causing it to fail to start, it is impossible to diagnose, find a solution, and test that solution.

The logs you want to find are from within Yarn UI. You will need to dig deep into the hive LLAP application logs, specifically looking for the failed container logs. It is usually 3-4 clicks deep into the UI pages where you will find several different log types. Some of those clicks are just a link to the # 1. So inspect all links. Inspecting all of these logs usually shows a very specific error or reason why LLAP fails to start. Researching these errors should match quite a few posts here in HCC or if you search in google even more results on other websites.

My other recommendation is to start with 1 LLAP container (you have 3 above). Get one container working and move up from there in small test loops. If you get into the Hive or LLAP tags here in the HCC you can find several posts with screen shots of the actual settings used to get LLAP working.

Additionally it sometimes takes my clusters a few startups to get LLAP running. Also sometimes ambari reports LLAP is not running when it is. Another restart in this situation usually resolves the ambari errors.

Re: Hive HiveServer2 Interactive LLAP doesn't start

New Contributor

I am also seeing this error. Can you please let me know how to fix?

Don't have an account?
Coming from Hortonworks? Activate your account here