Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Where does spark-submit look for Jar files?

avatar
Explorer

Hello,

 

I am running a sort job through spark-submit. I've also written a custom compression codec (instead of the standard lzf or snappy). The custom codec is packaged in a jar file. My goal is to have spark use my custom codec for compression.

 

For the first couple of compression operations, my codec is indeed used (I see debug messages). Later, there is an exception thrown (java.lang.NoSuchMethodException, stating that my Java class isn't found).

 

I'm using YARN. My jar file is on the master node. It's path is specified in the /etc/spark/conf/classpath.txt.

 

Any ideas on why it's not being found (and only sometimes)? Perhaps I ought to specify the jar file's location in some other way?

 

Your suggestions please.

 

Thanks.

 

 

10 REPLIES 10

avatar
New Contributor

I posted an issue yesterday that relates to this -- the spark-submit classpath seems to conflict with commons-compress from a suppiled uber-jar.

 

I've tried the --conf, --jar, and the --packages flags with spark-submit with no resolution.  

 

Spark 2.x + Tika: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamF

 

Any help would be greatly appreciated!!!!