question Re: library copy to /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib in Archives of Support Questions (Read Only)

library copy to /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib

Khirod — Fri, 16 Sep 2022 09:41:40 GMT

Hi,

I am very new to hadoop eco system . Please somebody help me to file(external jar from my local system) copy to all nodes(/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib) using cloudera manager . Not sure if we can do it through cloudera manager or we may have some other alternatives. Please suggest.

Regards

Khirod Patra

Re: library copy to /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib

GautamG — Thu, 24 Sep 2015 11:05:45 GMT

The easiest way is to build your own parcel. It would be best if your
custom jars are stored in a location outside of the CDH bits, this way
upgrades work smoothly. You can always add the location to
yarn.application.classpath
in Cloudera Manager, making it easier to support.

https://github.com/cloudera/cm_ext/wiki/Parcels:-What-and-Why%3F

Re: library copy to /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib

Khirod — Fri, 02 Oct 2015 06:40:38 GMT

Thank you Gautam,

I will try and update here how it goes.

-Khirod

Re: library copy to /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib

Khirod — Sun, 04 Oct 2015 16:04:04 GMT

Hi Goutham,

I tried with a custom parcel, but it would be always required some extra effort in upgrade as you suggested too, and it may be a caution to play with main cdh parcel. One more burden is , if I want to just add a required jar file in between , I have to create a new parcel again , which is not feasible. And please suggest if I do have some jar files only which I may need to copy to "/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib" only on each node . Let I have a cluster having 4 nodes , and I want all the jar file in "/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib" on each node.

Initially I thought to scp to each node, but seems this is not an effective way. So I guess, we may have some thing/or any exposed api just to transfer these files to cloudera manager and manager will takes care to put those files in respective place.

CLASS_PATH location is also good idea, but it is not specific to the YARN_SERVER only, I need this jar on each and every node, Please suggest and guide.

Regards
Khirod

Re: library copy to /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib

Khirod — Sun, 04 Oct 2015 16:17:49 GMT

One more thing, I think remote parcel would be helpful. Is there any cdh upgrade impact on this?

Thanks

Khirod

Re: library copy to /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib

Khirod — Wed, 07 Oct 2015 16:19:08 GMT

So Finaly remote parcel fullfill my requirements .

Thanks GauthamG for help and guide.

Regards

Khirod