Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

MR jobs calling 3rd party libs

avatar

I want to submit a job to Yarn, which has dependencies to third party libraries. Map reduce jobs fail as they can't load these third party libraries on remote Hadoop nodes. Can I use Yarn in someway, to submit a whole package (including 3rd party libs), so MR jobs can successfully load them?

1 ACCEPTED SOLUTION

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Mentor
@Koosha Tahmasebipour

you can leverage DistributedCache to upload your third party jars to HDFS https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/filecache/DistributedCache.html for MapReduce