question Re: MR jobs calling 3rd party libs in Archives of Support Questions (Read Only)

MR jobs calling 3rd party libs

koosha_tahmaseb — Fri, 09 Sep 2016 23:58:22 GMT

I want to submit a job to Yarn, which has dependencies to third party libraries. Map reduce jobs fail as they can't load these third party libraries on remote Hadoop nodes. Can I use Yarn in someway, to submit a whole package (including 3rd party libs), so MR jobs can successfully load them?

Re: MR jobs calling 3rd party libs

rajkumar_singh — Sat, 10 Sep 2016 00:01:01 GMT

did you try packaging all the 3rd party jars into uber jar and submit to cluster.

Re: MR jobs calling 3rd party libs

aervits — Sat, 10 Sep 2016 00:14:29 GMT

@Koosha Tahmasebipour

you can leverage DistributedCache to upload your third party jars to HDFS https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/filecache/DistributedCache.html for MapReduce