Reply
New Contributor
Posts: 4
Registered: ‎03-21-2017
Accepted Solution

Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIRST

Where do i update HADOOP_USER_CLASSPATH_FIRST=true and HADOOP_CLASSPATH in cloudera manager so that it has cluster wide effect.

 

I manually changed the seeting in /etc/hadoop/conf/hadoop-env.sh. However, i dont see that effect. Basically, i want to prepend classpath when starting hiveserver2 and metastore. This works fine on HDP when these parameters are set in hadoop-env.sh via ambari.

 

Cloudera Employee
Posts: 103
Registered: ‎03-23-2015

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

Try to go to:

CM > Hive > Configuration > search for "Hive Service Environment Advanced Configuration Snippet (Safety Valve)"

and put in environment settings you want to update here.

Cloudera Manager will then ask you to restart Hive service. This will affect both HS2 and HMS.
New Contributor
Posts: 4
Registered: ‎03-21-2017

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

[ Edited ]

I also want to override jars when i run mapreduce job. That is when i run wordcount example using hadoop as well as yarn. Where do i place these variables in cloudera manager ?

 

I tried following option:

CM > Yarn > Congihuration > search for "YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)"

 

This did not help. It still picks the old jars

Posts: 1,493
Kudos: 252
Solutions: 227
Registered: ‎07-31-2013

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

"Service Environment" applies to services, i.e. roles you run via CM. For
clients, use the equivalent "Client Environment" valve.

FWIW, the Cloudera Manager introduction page goes over the terminology it
uses to classify different layers of configuration:
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_intro_primer.html#concept_wfj_tny....
In your case, clients would mean gateways.

P.s. Typically a bad idea to override system jar versions, as the
software's been compiled with the version that was included along. Any
incompatible changes to methods or availability of classes within the
changed version you're forcing the classloader to pick would cause runtime
failures (NoSuchMethodError, NoClassDefFound, ClassNotFoundException, etc.).
P.p.s. If overriding some pre-included library is the prime purpose, try to
use Maven shading with namespace relocation:
https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
Backline Customer Operations Engineer
New Contributor
Posts: 4
Registered: ‎03-21-2017

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

Which client environment ?

 

I changed the following option:

CM > Yarn > Configuration > Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh

It has no effect. It is still picks the default library, not the one which i am trying to override. 

 

Service Environment Advanced Configuration works fine in case of hive.

New Contributor
Posts: 4
Registered: ‎03-21-2017

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

My bad. I forgot to deploy, stale configuration after making the changes.

CM > Yarn > Configuration > Gateway Client Environment Advanced Configuration Snippet (Safety Valve)  works fine. I am able to override the default jars

Posts: 1,493
Kudos: 252
Solutions: 227
Registered: ‎07-31-2013

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

That's the right one, but you'll need to perform a cluster-wide client
configuration redeploy to have it go to your local gateway configuration:
https://www.youtube.com/watch?v=4S9H3wftM_0
Backline Customer Operations Engineer
Announcements