question Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR in Archives of Support Questions (Read Only)

Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIRST

MangeshVC — Fri, 16 Sep 2022 11:18:19 GMT

Where do i update HADOOP_USER_CLASSPATH_FIRST=true and HADOOP_CLASSPATH in cloudera manager so that it has cluster wide effect.

I manually changed the seeting in /etc/hadoop/conf/hadoop-env.sh. However, i dont see that effect. Basically, i want to prepend classpath when starting hiveserver2 and metastore. This works fine on HDP when these parameters are set in hadoop-env.sh via ambari.

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

EricL — Thu, 23 Mar 2017 11:13:43 GMT

Try to go to:

CM > Hive > Configuration > search for "Hive Service Environment Advanced Configuration Snippet (Safety Valve)"

and put in environment settings you want to update here.

Cloudera Manager will then ask you to restart Hive service. This will affect both HS2 and HMS.

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

MangeshVC — Thu, 23 Mar 2017 16:36:14 GMT

I also want to override jars when i run mapreduce job. That is when i run wordcount example using hadoop as well as yarn. Where do i place these variables in cloudera manager ?

I tried following option:

CM > Yarn > Congihuration > search for "YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)"

This did not help. It still picks the old jars

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

Harsh J — Thu, 23 Mar 2017 16:48:50 GMT

"Service Environment" applies to services, i.e. roles you run via CM. For
clients, use the equivalent "Client Environment" valve.

FWIW, the Cloudera Manager introduction page goes over the terminology it
uses to classify different layers of configuration:
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_intro_primer.html#concept_wfj_tny_jk.
In your case, clients would mean gateways.

P.s. Typically a bad idea to override system jar versions, as the
software's been compiled with the version that was included along. Any
incompatible changes to methods or availability of classes within the
changed version you're forcing the classloader to pick would cause runtime
failures (NoSuchMethodError, NoClassDefFound, ClassNotFoundException, etc.).
P.p.s. If overriding some pre-included library is the prime purpose, try to
use Maven shading with namespace relocation:
https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

MangeshVC — Thu, 23 Mar 2017 17:10:34 GMT

Which client environment ?

I changed the following option:

CM > Yarn > Configuration > Gateway Client Environment Advanced Configuration Snippet (Safety Valve) for hadoop-env.sh

It has no effect. It is still picks the default library, not the one which i am trying to override.

Service Environment Advanced Configuration works fine in case of hive.

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

MangeshVC — Thu, 23 Mar 2017 17:21:05 GMT

My bad. I forgot to deploy, stale configuration after making the changes.

CM > Yarn > Configuration > Gateway Client Environment Advanced Configuration Snippet (Safety Valve) works fine. I am able to override the default jars

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

Harsh J — Thu, 23 Mar 2017 17:21:49 GMT

That's the right one, but you'll need to perform a cluster-wide client
configuration redeploy to have it go to your local gateway configuration:
https://www.youtube.com/watch?v=4S9H3wftM_0

Re: Configuration setting via Cloudera Manager to set HADOOP_CLASSPATH and HADOOP_USER_CLASSPATH_FIR

Nishan — Fri, 01 Feb 2019 22:39:17 GMT

@Harsh J How would I do this just for one job ?.

I tried using below setting but it is not working. The issue is that I want to use a version of jersey which I bundled into my fat jar,however gateway node has an older version of that jar and it loads a class from there resulting in a NosSuchMethodException .My application is not a map reduce job and I run it by using hadoop jar and running on 5.14.4

export HADOOP_USER_CLASSPATH_FIRST=true

export HADOOP_CLASSPATH=/projects/poc/test/config:$HADOOP_CLASSPATH