Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Where does YARN look for JAR files

Where does YARN look for JAR files

Contributor

I would like to create a new compression codec, (like LZF, LZ4), and specify the codec name in the YARN configuration menu: mapreduce.map.output.compress.codec.

 

My code is in Java. Where should I copy my new .jar file to, so that YARN can find it?

In other words, where does YARN look for the codec name that I specified above?

 

Thanks.

1 REPLY 1
Highlighted

Re: Where does YARN look for JAR files

Master Guru
The default classpath of MR2 application containers is controlled by the property "mapreduce.application.classpath" inside your mapred-site.xml. You can add other local paths that are available across the entire cluster to this config to extend it.

If you do not need this to be placed across the cluster, then you may pass it with the distributed cache's -libjars option (or code-equivalent).

For client-side classpath, HADOOP_CLASSPATH environment variable controls and allows addition of extra jars. This would require your jar to be on it for cases when the codec is used outside of a job runtime (such as in driver read/write stages).