Spark Testing base 1.6.1_0.3.3 for spark version 2.0.2 is throwing java.lang.NoClassDefFoundError: org/apache/spark/Logging

[#1]

A needed class was not found. This could be due to an error in your runpath. Missing class: org/apache/spark/Logging java.lang.NoClassDefFoundError: org/apache/spark/Logging

https://community.hortonworks.com/questions/58286/noclassdeffounderror-orgapachesparklogging-using-s...

Looked at the above question i think its valid for spark prior to 2.0 but iam looking for 2.0.2 🙂

I tried adding dependencies in sbt for Spark streaming, twitter and many more. Still i get this error for spark 2.0.2. Any ideas how to get this resolved? Thank you

[#2]

org,apache.spark.Logging was a private api in 2.0

It was always marked as a @Private, but was exposed till 1.6; and this was tightened in 2.0 to remove its visibility.

Bottomline is, you cant depend on it; and have to modify code to remove dependency on the trait..

Regards,

Mridul

[#3 · Accepted solution]

@Mridul M Thanks for the reply. Actually i was using the old version of spark testing base. Didn't know there was a 2.0.2 version. Holden karau pointed me to maven central where i found the latest version of spark testing base. In github readme of spark testing base it was mentioned 1.6 so i assumed the latest version was 1.6. But now its sorted and spark 2.0.2 Dataframe testing works for me. In case people might need this it also needs Hive dependency. Currently its new and couldnt find forums in spark 2.0 testing so iam posting this might help save time for other developers 🙂