question Re: Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace in Support Questions

Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace

Nandyal — Fri, 16 Sep 2022 14:24:16 GMT

Despite adding the following,

--conf spark.driver.extraClassPath=/opt/cloudera/parcels/CDH/jars/htrace-core-3.2.0-incubating.jar:/opt/cloudera/parcels/CDH/lib/hive/conf:/opt/cloudera/parcels/CDH/lib/hive/lib/*.jar \
--conf spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/jars/htrace-core-3.2.0-incubating.jar:/opt/cloudera/parcels/CDH/lib/hive/conf:/opt/cloudera/parcels/CDH/lib/hive/lib/*.jar \

I get the following exception:

Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace

I have checked the file permission and directory permission and they have the right permission (directories have and read and execute perms while the files have 755 perms).

Please let me know what needs to be changed.

Re: Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace

AKR — Mon, 27 May 2019 17:13:08 GMT

Hi Nandyal,

From the Error it seems there was an jar file missing. Could you please share the entire stacktrace for further analysis on this issue?

Thanks

Re: Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace

Nandyal — Tue, 28 May 2019 16:15:57 GMT

Hey AK,

Following is the stack trace:

10:13:28,194 WARN [TaskSetManager] Lost task 8.0 in stage 1.0 (TID 4, hostname.domain.com, executor 12): java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
at org.apache.hadoop.hbase.spark.HBaseConnectionCache$$anonfun$getConnection$1.apply(HBaseConnectionCache.scala:121)
at org.apache.hadoop.hbase.spark.HBaseConnectionCache$$anonfun$getConnection$1.apply(HBaseConnectionCache.scala:121)
at org.apache.hadoop.hbase.spark.HBaseConnectionCache$$anonfun$1.apply(HBaseConnectionCache.scala:114)
at org.apache.hadoop.hbase.spark.HBaseConnectionCache$$anonfun$1.apply(HBaseConnectionCache.scala:113)
at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
at org.apache.hadoop.hbase.spark.HBaseConnectionCache$.getConnection(HBaseConnectionCache.scala:113)
at org.apache.hadoop.hbase.spark.HBaseConnectionCache$.getConnection(HBaseConnectionCache.scala:121)
at org.apache.hadoop.hbase.spark.HBaseContext.org$apache$hadoop$hbase$spark$HBaseContext$$hbaseForeachPartition(HBaseContext.scala:459)
at org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$bulkPut$1.apply(HBaseContext.scala:217)
at org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$bulkPut$1.apply(HBaseContext.scala:217)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$33.apply(RDD.scala:920)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
... 23 more
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:216)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:419)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:935)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:659)
... 28 more
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 34 more

Re: Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace

Nandyal — Tue, 28 May 2019 16:18:42 GMT

And the Trace class is present in htrace-core-3.2.0-incubating.jar, which has been set in

spark.driver.extraclasspath and spark.executor.extraClassPath config variables at the time of spark-submit.

Thank you!

Nandyal

Re: Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace

Nandyal — Tue, 04 Jun 2019 14:45:09 GMT

I tried doing this as well, with no success:
added the hbase jars to the executor classpath via the following setting:
signed in to ClouderaManager
went to the Spark on YARN service
went to the Configuration tab
typed defaults in the search box
selected gateway in the scope
added the entry:
spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.2.0-incubating.jar

Re: Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace

Nandyal — Wed, 12 Jun 2019 15:07:14 GMT

After making a small change to the location of the jar, we got it working. The steps are as follows:

added the hbase jars to the executor classpath via the following setting:
signed in to ClouderaManager
went to the Spark on YARN service
went to the Configuration tab
typed defaults in the search box
selected gateway in the scope
added the entry:
spark.executor.extraClassPath=/hdfs03/parcels/CDH/lib/hbase/lib/htrace-core-3.2.0-incubating.jar

Note: we had to use the hdfs directory path.