Where does spark-submit look for Jar files?

[#1]

Hello,

I am running a sort job through spark-submit. I've also written a custom compression codec (instead of the standard lzf or snappy). The custom codec is packaged in a jar file. My goal is to have spark use my custom codec for compression.

For the first couple of compression operations, my codec is indeed used (I see debug messages). Later, there is an exception thrown (java.lang.NoSuchMethodException, stating that my Java class isn't found).

I'm using YARN. My jar file is on the master node. It's path is specified in the /etc/spark/conf/classpath.txt.

Any ideas on why it's not being found (and only sometimes)? Perhaps I ought to specify the jar file's location in some other way?

Your suggestions please.

Thanks.

[#2]

I posted an issue yesterday that relates to this -- the spark-submit classpath seems to conflict with commons-compress from a suppiled uber-jar.

I've tried the --conf, --jar, and the --packages flags with spark-submit with no resolution.  

Spark 2.x + Tika: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamF

Any help would be greatly appreciated!!!!