Support Questions
Find answers, ask questions, and share your expertise

MR jobs calling 3rd party libs

Solved Go to solution
Highlighted

MR jobs calling 3rd party libs

I want to submit a job to Yarn, which has dependencies to third party libraries. Map reduce jobs fail as they can't load these third party libraries on remote Hadoop nodes. Can I use Yarn in someway, to submit a whole package (including 3rd party libs), so MR jobs can successfully load them?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: MR jobs calling 3rd party libs

did you try packaging all the 3rd party jars into uber jar and submit to cluster.

View solution in original post

2 REPLIES 2
Highlighted

Re: MR jobs calling 3rd party libs

did you try packaging all the 3rd party jars into uber jar and submit to cluster.

View solution in original post

Highlighted

Re: MR jobs calling 3rd party libs

Mentor
@Koosha Tahmasebipour

you can leverage DistributedCache to upload your third party jars to HDFS https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/filecache/DistributedCache.html for MapReduce

Don't have an account?