Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Adding hadoop jars to CLASSPATH in CSD/Parcel

Adding hadoop jars to CLASSPATH in CSD/Parcel

Explorer

Hi All,
I'm working with CDH-5.3.3 and trying to build a parcle/CSD.  Our services need to build up a classpath with all the hadoop jars on it - we typically do that by spinning through HADOOP_HOME/*.jar, ./lib/*.jar, etc in a config script[1].  However, CM HADOOP_HOME / CDH_HADOOP_HOME seems pretty useless for those purposes since they're pretty well empty.  I sourced the $COMMON_SCRIPT to call 'set_hadoop_classpath' but it the HADOOP_CLASSPATH that it creates is also pretty useless. 

Poking around the parces leads me to believe that I could build it on something like $PARCELS_ROOT/CDH/lib/hadoop... but that doesn't feel right.  So, I wonder what the idiomatic way to snag the location of various hadoop jars and add them to the classpath of a service is?  The way that Accumulo's is written[2] makes me believe that HADOOP_HOME may usually be something useful though, so perhaps something's misconfigured on my end?

Thanks,
--tim

[1] - https://github.com/apache/incubator-blur/blob/master/distribution/src/main/scripts/bin/blur-config.s...
[2] - https://github.com/cloudera/cm_csds/blob/master/ACCUMULO16/src/scripts/accumulo.sh

3 REPLIES 3

Re: Adding hadoop jars to CLASSPATH in CSD/Parcel

Explorer
So, I suppose a more succinct question would be... how do custom CSD control scripts get access to HADOOP_PREFIX? Are they required to export it themselves in their corresponding parcel?
Highlighted

Re: Adding hadoop jars to CLASSPATH in CSD/Parcel

Cloudera Employee

@williamstw wrote:
So, I suppose a more succinct question would be... how do custom CSD control scripts get access to HADOOP_PREFIX? Are they required to export it themselves in their corresponding parcel?

I would, and do, recommend you simply use ../CDH/lib/... As long as your parcel depends on CDH, then you can be sure that CDH exists and that it is present in this location. Parcels are required to name their directories the same as their logical names, so this is safe and consistent behaviour.

Re: Adding hadoop jars to CLASSPATH in CSD/Parcel

New Contributor

This is my question too:

How do custom CSD control scripts get access to HADOOP_PREFIX? Are they required to export it themselves in their corresponding parcel?

I love idioms and Cloudera.
Don't have an account?
Coming from Hortonworks? Activate your account here