Hiveserver2 Interactive (LLAP) doesn't start

[#1]

Hello Team,

We are using HDP-2.6.5 in our environment.

We are getting following error while starting up Hiveserver2 Interactive (LLAP) server:

2019-03-27T04:33:05,291 ERROR [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception
java.lang.NoClassDefFoundError: org/apache/tez/hadoop/shim/HadoopShimsLoader
    at org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.<init>(ContainerRunnerImpl.java:157) ~[hive-llap-server-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
    at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.<init>(LlapDaemon.java:291) ~[hive-llap-server-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
    at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
Caused by: java.lang.ClassNotFoundException: org.apache.tez.hadoop.shim.HadoopShimsLoader
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_192]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_192]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_192]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_192]
    ... 3 more

On all the nodes of cluster, hadoop-shim* jars are present as given below:

[root@p-hdp-01 ~]# cd /usr/hdp/2.6.5.0-292/tez_hive2/
[root@p-hdp-01 tez_hive2]# ls -lrth
total 4.0M
-rw-r--r-- 1 root root  23K May 11  2018 tez-yarn-timeline-history-with-fs-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 7.7K May 11  2018 tez-yarn-timeline-history-with-acls-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root  28K May 11  2018 tez-yarn-timeline-history-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root  12K May 11  2018 tez-yarn-timeline-cache-plugin-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 155K May 11  2018 tez-tests-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 642K May 11  2018 tez-runtime-library-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 192K May 11  2018 tez-runtime-internals-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 284K May 11  2018 tez-mapreduce-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root  72K May 11  2018 tez-job-analyzer-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root  15K May 11  2018 tez-javadoc-tools-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root  77K May 11  2018 tez-history-parser-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 106K May 11  2018 tez-ext-service-tests-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root  56K May 11  2018 tez-examples-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.3M May 11  2018 tez-dag-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root  76K May 11  2018 tez-common-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 975K May 11  2018 tez-api-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 5.4K May 11  2018 hadoop-shim-hdp-0.8.4.2.6.5.0-292.jar
-rw-r--r-- 1 root root 8.7K May 11  2018 hadoop-shim-0.8.4.2.6.5.0-292.jar
drwxr-xr-x 2 root root    6 May 11  2018 doc
lrwxrwxrwx 1 root root   19 Mar 25 06:16 conf -> /etc/tez_hive2/conf
drwxr-xr-x 3 root root   18 Mar 25 06:16 man
drwxr-xr-x 2 root root 4.0K Mar 25 06:16 lib
drwxr-xr-x 2 root root   42 Mar 25 06:16 ui
[root@p-hdp-01 tez_hive2]

How to resolve this?

Please suggest.

Thanks,

Bhushan

[#2]

@Nitin Shelke @Jay Kumar SenSharma..please suggest.

[#3]

@Sergey Shelukhin @gopal @Dennis Connolly @Mahmoud Sabri .... please suggest

[#4]

@Bhushan Kandalkar I guess these files are present, but still LLAP is not able to pick the aux jars. Can you please try to add these jars into AUX jars and try start of LLAP. You can try to add them like below,

- Add below 2 files in Advanced hive-interactive-env -> Auxillary JAR list
/usr/hdp/2.6.5.0-292/tez_hive2/hadoop-shim-hdp-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/hadoop-shim-0.8.4.2.6.5.0-292.jar

Thanks

Nitin Shelke

[#5]

Thanks @Nitin Shelke. This resolved LLAP starting issue. But while inserting any data in Hive table using LLAP we got below error:

TaskAttempt 3 failed, info=[org.apache.hadoop.ipc.RemoteException(java.lang.NoClassDefFoundError): org/apache/tez/runtime/internals/api/TaskReporterInterface
        at org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:263)
        at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:554)
        at org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:101)
        at org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:16818)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
Caused by: java.lang.ClassNotFoundException: org.apache.tez.runtime.internals.api.TaskReporterInterface
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 12 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1553784387057_0006_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2)

To resolve this issue, added all jars under '/usr/hdp/2.6.5.0-292/tez_hive2' location in Auxilary Jar list as follow:

/usr/hdp/2.6.5.0-292/tez_hive2/hadoop-shim-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/hadoop-shim-hdp-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-api-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-common-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-dag-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-examples-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-ext-service-tests-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-history-parser-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-javadoc-tools-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-job-analyzer-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-mapreduce-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-runtime-internals-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-runtime-library-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-tests-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-yarn-timeline-cache-plugin-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-yarn-timeline-history-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-yarn-timeline-history-with-acls-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-yarn-timeline-history-with-fs-0.8.4.2.6.5.0-292.jar

Now, able to insert data in Hive table.