Reply
Explorer
Posts: 14
Registered: ‎06-05-2016

NoClassDefFoundError in Oryx batch and speed layers

Hi guys,

 

have you seen this error before? I am trying to run Oryx.

17/08/26 12:23:16 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
17/08/26 12:23:16 INFO cluster.YarnClientSchedulerBackend: Stopped
17/08/26 12:23:16 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/08/26 12:23:16 INFO memory.MemoryStore: MemoryStore cleared
17/08/26 12:23:16 INFO storage.BlockManager: BlockManager stopped
17/08/26 12:23:16 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
17/08/26 12:23:16 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/08/26 12:23:16 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError: kafka/admin/RackAwareMode
        at com.cloudera.oryx.lambda.AbstractSparkLayer.buildInputDStream(AbstractSparkLayer.java:179)
        at com.cloudera.oryx.lambda.speed.SpeedLayer.start(SpeedLayer.java:97)
        at com.cloudera.oryx.speed.Main.main(Main.java:33)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: kafka.admin.RackAwareMode
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 12 more
17/08/26 12:23:16 INFO util.ShutdownHookManager: Shutdown hook called
17/08/26 12:23:16 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-29ab2cba-79a0-4a19-8663-eb976d2b43eb

We're using

- CDH-5.12.0-1.cdh5.12.0.p0.29

SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354

- Confluent Platform 3.3 as a Kafka Cluster (Kafka version is 0.11.0.0)

 

Can Kafka version 0.11.0.0 be a problem? There should be no change in message format between 0.10.x and 0.11.x versions afaik. If thats the cause, can you point me how to add support for 0.11.x to Oryx? I will try to open pull-request :).

 

Thanks in advance for any suggestion.

 

Sincerely,

Matus Cimerman

Cloudera Employee
Posts: 447
Registered: ‎08-11-2014

Re: NoClassDefFoundError in Oryx batch and speed layers

Yeah, this suggests a Kafka version problem. Kafka is notoriously incompatible across even minor versions. I would suspect however that the latest 2.5.x release of Oryx, which compiles vs 0.10.2+, does work with 0.11.x. But you might have to compile your own build if you're using Kafka 0.11.x or 1.x.

Explorer
Posts: 14
Registered: ‎06-05-2016

Re: NoClassDefFoundError in Oryx batch and speed layers

I'm using 0.11.x. When will be 2.5.x Oryx release available?
Explorer
Posts: 14
Registered: ‎06-05-2016

Re: NoClassDefFoundError in Oryx batch and speed layers

Hi Sean,

 

is there any way how can I help you with new 2.5.x Oryx release?

Cloudera Employee
Posts: 447
Registered: ‎08-11-2014

Re: NoClassDefFoundError in Oryx batch and speed layers

If you've got a need for it I can just cut a release. If you have the opportunity to test against a build from master, all the better, as you can verify it works for you. If you're using Kafka after 0.10.2, I'm not sure it will work though, so just confirm.

Explorer
Posts: 14
Registered: ‎06-05-2016

Re: NoClassDefFoundError in Oryx batch and speed layers

Sure thing. I can test it against build from master - is it available somewhere or should I build it by myself?

Announcements