Created 03-27-2019 10:56 AM
Hello Team,
We are using HDP-2.6.5 in our environment.
We are getting following error while starting up Hiveserver2 Interactive (LLAP) server:
2019-03-27T04:33:05,291 ERROR [main ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception java.lang.NoClassDefFoundError: org/apache/tez/hadoop/shim/HadoopShimsLoader at org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.<init>(ContainerRunnerImpl.java:157) ~[hive-llap-server-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.<init>(LlapDaemon.java:291) ~[hive-llap-server-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292] Caused by: java.lang.ClassNotFoundException: org.apache.tez.hadoop.shim.HadoopShimsLoader at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_192] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_192] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_192] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_192] ... 3 more
On all the nodes of cluster, hadoop-shim* jars are present as given below:
[root@p-hdp-01 ~]# cd /usr/hdp/2.6.5.0-292/tez_hive2/ [root@p-hdp-01 tez_hive2]# ls -lrth total 4.0M -rw-r--r-- 1 root root 23K May 11 2018 tez-yarn-timeline-history-with-fs-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 7.7K May 11 2018 tez-yarn-timeline-history-with-acls-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 28K May 11 2018 tez-yarn-timeline-history-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 12K May 11 2018 tez-yarn-timeline-cache-plugin-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 155K May 11 2018 tez-tests-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 642K May 11 2018 tez-runtime-library-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 192K May 11 2018 tez-runtime-internals-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 284K May 11 2018 tez-mapreduce-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 72K May 11 2018 tez-job-analyzer-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 15K May 11 2018 tez-javadoc-tools-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 77K May 11 2018 tez-history-parser-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 106K May 11 2018 tez-ext-service-tests-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 56K May 11 2018 tez-examples-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 1.3M May 11 2018 tez-dag-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 76K May 11 2018 tez-common-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 975K May 11 2018 tez-api-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 5.4K May 11 2018 hadoop-shim-hdp-0.8.4.2.6.5.0-292.jar -rw-r--r-- 1 root root 8.7K May 11 2018 hadoop-shim-0.8.4.2.6.5.0-292.jar drwxr-xr-x 2 root root 6 May 11 2018 doc lrwxrwxrwx 1 root root 19 Mar 25 06:16 conf -> /etc/tez_hive2/conf drwxr-xr-x 3 root root 18 Mar 25 06:16 man drwxr-xr-x 2 root root 4.0K Mar 25 06:16 lib drwxr-xr-x 2 root root 42 Mar 25 06:16 ui [root@p-hdp-01 tez_hive2]
How to resolve this?
Please suggest.
Thanks,
Bhushan
Created 03-28-2019 05:46 AM
@Nitin Shelke @Jay Kumar SenSharma..please suggest.
Created 03-28-2019 01:27 PM
@Sergey Shelukhin @gopal @Dennis Connolly @Mahmoud Sabri .... please suggest
Created 03-29-2019 04:25 PM
@Bhushan Kandalkar I guess these files are present, but still LLAP is not able to pick the aux jars. Can you please try to add these jars into AUX jars and try start of LLAP. You can try to add them like below,
- Add below 2 files in Advanced hive-interactive-env -> Auxillary JAR list
/usr/hdp/2.6.5.0-292/tez_hive2/hadoop-shim-hdp-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/hadoop-shim-0.8.4.2.6.5.0-292.jar
Thanks
Nitin Shelke
Created 03-30-2019 08:03 AM
Thanks @Nitin Shelke. This resolved LLAP starting issue. But while inserting any data in Hive table using LLAP we got below error:
TaskAttempt 3 failed, info=[org.apache.hadoop.ipc.RemoteException(java.lang.NoClassDefFoundError): org/apache/tez/runtime/internals/api/TaskReporterInterface at org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:263) at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:554) at org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:101) at org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:16818) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) Caused by: java.lang.ClassNotFoundException: org.apache.tez.runtime.internals.api.TaskReporterInterface at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 12 more ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1553784387057_0006_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2)
To resolve this issue, added all jars under '/usr/hdp/2.6.5.0-292/tez_hive2' location in Auxilary Jar list as follow:
/usr/hdp/2.6.5.0-292/tez_hive2/hadoop-shim-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/hadoop-shim-hdp-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-api-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-common-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-dag-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-examples-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-ext-service-tests-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-history-parser-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-javadoc-tools-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-job-analyzer-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-mapreduce-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-runtime-internals-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-runtime-library-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-tests-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-yarn-timeline-cache-plugin-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-yarn-timeline-history-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-yarn-timeline-history-with-acls-0.8.4.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/tez_hive2/tez-yarn-timeline-history-with-fs-0.8.4.2.6.5.0-292.jar
Now, able to insert data in Hive table.