question On CDH5.3.2, artifacts lib jar precedence over Cloudera Jars in Archives of Support Questions (Read Only)

On CDH5.3.2, artifacts lib jar precedence over Cloudera Jars

ppatel11 — Fri, 16 Sep 2022 09:25:08 GMT

On CDH5.3.2, classpath for artifacts lib jar does not precedes over Cloudera Jars. Our artifact uses (aws-java-sdk-1.7.12.jar) which is part of artifact Lib dir. However Cloudera loads a previous version jar (aws-java-sdk-1.7.4.jar)

Current Jar loading Precedence:

1) /opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/aws-java-sdk-1.7.4.jar

2) /data3/yarn/nm/usercache/oozie/appcache/application_.../container_.../aws-java-sdk-1.7.12.jar

This throws throws java.lang.NoSuchMethodError

Desired Jar Loading Precedence:

1) /data3/yarn/nm/usercache/oozie/appcache/application_.../container_.../aws-java-sdk-1.7.12.jar

2) /opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/aws-java-sdk-1.7.4.jar

Tried vairous online suggested property file changes to 'true'.

* oozie.launcher.mapreduce.task.classpath.user.precedence

* mapreduce.task.classpath.user.precedence
* mapreduce.task.classpath.first
* mapreduce.job.user.classpath.first
* mapreduce.user.classpath.first

* mapreduce.task.classpath.user.precedence

However all the above in the job.properties and workflow file did not change the precedence. I working with a Java action here.

This feature was easily reverted in previous CHD4+ releases.

Please help.

Re: On CDH5.3.2, artifacts lib jar precedence over Cloudera Jars

ppatel11 — Wed, 25 Mar 2015 04:35:38 GMT

Was able to fix this however had few hickups. Here are the details
1) Set oozie.launcher.mapreduce.task.classpath.user.precedence to true
2) Precedene was alterd. Howerever encountered:

--------------------------------------------------------------------------------------------
Application application_1426549201832_0261 failed 2 times due to AM Container for appattempt_1426549201832_0261_000002 exited with exitCode: 1 due to: Exception from container-launch.
Container id: container_1426549201832_0261_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.

2015-03-24 15:43:30,899 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.TaskLog.createLogSyncer()Ljava/util/concurrent/ScheduledExecutorService;
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:244)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:227)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1390)
2015-03-24 15:43:30,913 INFO [main]
org.apache.hadoop.util.ExitUtil: Exiting with status 1
--------------------------------------------------------------------------------------------

Suggesting... Jars found in the artifact lib for hadoop-code were in conflict with the expected jars by the CDH5.3.2 cluster we had.

3) Hadoop-core jars from Artifacts POM were excluded from the artifact package....
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<scope>provided</scope>
</dependency>

Things started to work as expected hereafter.

Re: On CDH5.3.2, artifacts lib jar precedence over Cloudera Jars

nur.majid — Tue, 16 Jun 2015 12:04:09 GMT

Hi,

I applied point 1 with following syntax, and still not work:

$ hadoop jar JarName.jar ClassName -Doozie.launcher.mapreduce.task.classpath.user.precedence=true param1 param2

In my case, I have fat-jar with modified parquet lib in it. But cluster always picking default parquet lib in CDH. In standalone mode, my program is running fine.

Re: On CDH5.3.2, artifacts lib jar precedence over Cloudera Jars

Wilfred — Wed, 17 Jun 2015 01:14:55 GMT

Using the class path precedence is not the correct solution for all cases.

A solution that will work in all cases is to use shading for the classes that you have modified versions of (use maven or gradle to do that).

In your case you need to shade the parquet classes that you have modified when you package the jar.

Be careful if you change classes like parquet: you could ed up with files that are only readable with your code and force you to keep packaging it with all jobs. That could cause problems later if you decide to use a different method to access the files.

Wilfred