Support Questions

Find answers, ask questions, and share your expertise

MR jobs calling 3rd party libs

avatar

I want to submit a job to Yarn, which has dependencies to third party libraries. Map reduce jobs fail as they can't load these third party libraries on remote Hadoop nodes. Can I use Yarn in someway, to submit a whole package (including 3rd party libs), so MR jobs can successfully load them?

1 ACCEPTED SOLUTION

avatar
Super Guru

did you try packaging all the 3rd party jars into uber jar and submit to cluster.

View solution in original post

2 REPLIES 2

avatar
Super Guru

did you try packaging all the 3rd party jars into uber jar and submit to cluster.

avatar
Master Mentor
@Koosha Tahmasebipour

you can leverage DistributedCache to upload your third party jars to HDFS https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/filecache/DistributedCache.html for MapReduce