Reply
Explorer
Posts: 14
Registered: ‎06-05-2016

NoClassDefFoundError in Oryx batch and speed layers

Hi guys,

 

have you seen this error before? I am trying to run Oryx.

17/08/26 12:23:16 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
17/08/26 12:23:16 INFO cluster.YarnClientSchedulerBackend: Stopped
17/08/26 12:23:16 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/08/26 12:23:16 INFO memory.MemoryStore: MemoryStore cleared
17/08/26 12:23:16 INFO storage.BlockManager: BlockManager stopped
17/08/26 12:23:16 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
17/08/26 12:23:16 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/08/26 12:23:16 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError: kafka/admin/RackAwareMode
        at com.cloudera.oryx.lambda.AbstractSparkLayer.buildInputDStream(AbstractSparkLayer.java:179)
        at com.cloudera.oryx.lambda.speed.SpeedLayer.start(SpeedLayer.java:97)
        at com.cloudera.oryx.speed.Main.main(Main.java:33)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: kafka.admin.RackAwareMode
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 12 more
17/08/26 12:23:16 INFO util.ShutdownHookManager: Shutdown hook called
17/08/26 12:23:16 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-29ab2cba-79a0-4a19-8663-eb976d2b43eb

We're using

- CDH-5.12.0-1.cdh5.12.0.p0.29

SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354

- Confluent Platform 3.3 as a Kafka Cluster (Kafka version is 0.11.0.0)

 

Can Kafka version 0.11.0.0 be a problem? There should be no change in message format between 0.10.x and 0.11.x versions afaik. If thats the cause, can you point me how to add support for 0.11.x to Oryx? I will try to open pull-request :).

 

Thanks in advance for any suggestion.

 

Sincerely,

Matus Cimerman

Cloudera Employee
Posts: 453
Registered: ‎08-11-2014

Re: NoClassDefFoundError in Oryx batch and speed layers

Yeah, this suggests a Kafka version problem. Kafka is notoriously incompatible across even minor versions. I would suspect however that the latest 2.5.x release of Oryx, which compiles vs 0.10.2+, does work with 0.11.x. But you might have to compile your own build if you're using Kafka 0.11.x or 1.x.

Explorer
Posts: 14
Registered: ‎06-05-2016

Re: NoClassDefFoundError in Oryx batch and speed layers

I'm using 0.11.x. When will be 2.5.x Oryx release available?
Explorer
Posts: 14
Registered: ‎06-05-2016

Re: NoClassDefFoundError in Oryx batch and speed layers

Hi Sean,

 

is there any way how can I help you with new 2.5.x Oryx release?

Cloudera Employee
Posts: 453
Registered: ‎08-11-2014

Re: NoClassDefFoundError in Oryx batch and speed layers

If you've got a need for it I can just cut a release. If you have the opportunity to test against a build from master, all the better, as you can verify it works for you. If you're using Kafka after 0.10.2, I'm not sure it will work though, so just confirm.

Explorer
Posts: 14
Registered: ‎06-05-2016

Re: NoClassDefFoundError in Oryx batch and speed layers

Sure thing. I can test it against build from master - is it available somewhere or should I build it by myself?

New Contributor
Posts: 6
Registered: ‎10-02-2017

Re: NoClassDefFoundError in Oryx batch and speed layers

Sean,

 

 I got the same error. I am using kafka 0.11.0.1 and Oryx 2.5.

 I am using AWS EMR (hadoop 2.7.3; spark 2.2.0 and zk 3.4.10)

 It looks cannot locate kafka related jar file. Is any way to pass in the jar file ?

 Thanks.

 

=======

17/10/05 02:25:17 INFO YarnClientSchedulerBackend: Shutting down all executors
17/10/05 02:25:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
17/10/05 02:25:17 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
17/10/05 02:25:17 INFO YarnClientSchedulerBackend: Stopped
17/10/05 02:25:17 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/10/05 02:25:17 INFO MemoryStore: MemoryStore cleared
17/10/05 02:25:17 INFO BlockManager: BlockManager stopped
17/10/05 02:25:17 INFO BlockManagerMaster: BlockManagerMaster stopped
17/10/05 02:25:17 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/10/05 02:25:17 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError: kafka/admin/RackAwareMode
at com.cloudera.oryx.lambda.AbstractSparkLayer.buildInputDStream(AbstractSparkLayer.java:179)
at com.cloudera.oryx.lambda.batch.BatchLayer.start(BatchLayer.java:105)
at com.cloudera.oryx.batch.Main.main(Main.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: kafka.admin.RackAwareMode
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more
17/10/05 02:25:17 INFO ShutdownHookManager: Shutdown hook called
17/10/05 02:25:17 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-73fdbc88-0da1-498e-aece-7abc3856aad1

Highlighted
Cloudera Employee
Posts: 453
Registered: ‎08-11-2014

Re: NoClassDefFoundError in Oryx batch and speed layers

That class was added in Kafka 0.10, so normally I'd say you have a version mismatch, and you're using a version of Oryx built for Kafka 0.10+ with older Kafka. However you say you're using Kafka 0.11. But I think the issue is that something else is bringing older Kafka client libs onto your classpath.

 

For example, I think the Spark examples JAR file brings it in? that could be a source of old Kafka 0.8 libs. Or somehow the classpath EMR sets up is bringing in Spark's Kafka 0.8 and 0.10 integration or something?

 

This part is definitely tricky. I know it works on CDH, mostly because its Spark 2 distro only supports Kafka 0.10.

Announcements