Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark Testing base 1.6.1_0.3.3 for spark version 2.0.2 is throwing java.lang.NoClassDefFoundError: org/apache/spark/Logging

avatar
Expert Contributor

A needed class was not found. This could be due to an error in your runpath. Missing class: org/apache/spark/Logging java.lang.NoClassDefFoundError: org/apache/spark/Logging

https://community.hortonworks.com/questions/58286/noclassdeffounderror-orgapachesparklogging-using-s...

Looked at the above question i think its valid for spark prior to 2.0 but iam looking for 2.0.2 🙂

I tried adding dependencies in sbt for Spark streaming, twitter and many more. Still i get this error for spark 2.0.2. Any ideas how to get this resolved? Thank you

1 ACCEPTED SOLUTION

avatar
Expert Contributor

@Mridul M Thanks for the reply. Actually i was using the old version of spark testing base. Didn't know there was a 2.0.2 version. Holden karau pointed me to maven central where i found the latest version of spark testing base. In github readme of spark testing base it was mentioned 1.6 so i assumed the latest version was 1.6. But now its sorted and spark 2.0.2 Dataframe testing works for me. In case people might need this it also needs Hive dependency. Currently its new and couldnt find forums in spark 2.0 testing so iam posting this might help save time for other developers 🙂

View solution in original post

2 REPLIES 2

avatar

org,apache.spark.Logging was a private api in 2.0

It was always marked as a @Private, but was exposed till 1.6; and this was tightened in 2.0 to remove its visibility.

Bottomline is, you cant depend on it; and have to modify code to remove dependency on the trait..

Regards,

Mridul

avatar
Expert Contributor

@Mridul M Thanks for the reply. Actually i was using the old version of spark testing base. Didn't know there was a 2.0.2 version. Holden karau pointed me to maven central where i found the latest version of spark testing base. In github readme of spark testing base it was mentioned 1.6 so i assumed the latest version was 1.6. But now its sorted and spark 2.0.2 Dataframe testing works for me. In case people might need this it also needs Hive dependency. Currently its new and couldnt find forums in spark 2.0 testing so iam posting this might help save time for other developers 🙂