library copy to /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib

[#1]

Hi,

I am very new to hadoop eco system . Please somebody help me to file(external jar from my local system) copy to all nodes(/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib) using cloudera manager . Not sure if we can do it through cloudera manager  or we may have some other alternatives. Please suggest.

Regards

Khirod Patra

[#2]

The easiest way is to build your own parcel. It would be best if your
custom jars are stored in a location outside of the CDH bits, this way
upgrades work smoothly. You can always add the location to
yarn.application.classpath
in Cloudera Manager, making it easier to support.

https://github.com/cloudera/cm_ext/wiki/Parcels:-What-and-Why%3F

[#3]

Thank you Gautam,

I will try and update here how it goes.

-Khirod

[#4]

Hi Goutham,

I tried with a custom parcel, but it would be always required some extra effort in upgrade as you suggested too, and it may be a caution to play with main cdh parcel. One more burden is , if I want to just add a required jar file in between , I have to create a new parcel again , which is not feasible. And please suggest if I do have some jar files only which I may need to copy  to "/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib" only on each node . Let I have a cluster having 4 nodes , and I want all the jar file in "/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib" on each node. 

Initially I thought to scp to each node, but seems this is not an effective way. So I guess, we may have some thing/or any exposed api  just to transfer these files to cloudera manager and manager will takes care to put those files in respective place.

CLASS_PATH location is also good idea, but it is not specific to the YARN_SERVER only, I need this jar on each and every node, Please suggest and guide.

Regards
Khirod

[#5]

One more thing, I think remote parcel would be helpful. Is there any cdh upgrade impact on this?

Thanks

Khirod

[#6 · Accepted solution]

So Finaly remote parcel fullfill my requirements .

Thanks GauthamG for help and guide.

Regards

Khirod