On CDH5.3.2, artifacts lib jar precedence over Cloudera Jars

[#1]

On CDH5.3.2, classpath for artifacts lib jar does not precedes over Cloudera Jars. Our artifact uses (aws-java-sdk-1.7.12.jar) which is part of artifact Lib dir. However Cloudera loads a previous version jar  (aws-java-sdk-1.7.4.jar) 

Current Jar loading Precedence:

1) /opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/aws-java-sdk-1.7.4.jar

2) /data3/yarn/nm/usercache/oozie/appcache/application_.../container_.../aws-java-sdk-1.7.12.jar

This throws  throws  java.lang.NoSuchMethodError

Desired Jar Loading Precedence:

1) /data3/yarn/nm/usercache/oozie/appcache/application_.../container_.../aws-java-sdk-1.7.12.jar

2) /opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/aws-java-sdk-1.7.4.jar

Tried vairous online suggested property file changes to 'true'.

* oozie.launcher.mapreduce.task.classpath.user.precedence

* mapreduce.task.classpath.user.precedence
* mapreduce.task.classpath.first
* mapreduce.job.user.classpath.first
* mapreduce.user.classpath.first

* mapreduce.task.classpath.user.precedence

However all the above in the job.properties and workflow file did not change the precedence. I working with a  Java action here.

This feature was easily reverted in previous CHD4+ releases.

Please help.

[#2 · Accepted solution]

Was able to fix this however had few hickups. Here are the details
1) Set oozie.launcher.mapreduce.task.classpath.user.precedence to true
2) Precedene was alterd. Howerever encountered:

--------------------------------------------------------------------------------------------
Application application_1426549201832_0261 failed 2 times due to AM Container for appattempt_1426549201832_0261_000002 exited with exitCode: 1 due to: Exception from container-launch.
Container id: container_1426549201832_0261_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.

2015-03-24 15:43:30,899 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.TaskLog.createLogSyncer()Ljava/util/concurrent/ScheduledExecutorService;
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:244)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:227)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1390)
2015-03-24 15:43:30,913 INFO [main]
org.apache.hadoop.util.ExitUtil: Exiting with status 1
--------------------------------------------------------------------------------------------

Suggesting... Jars found in the artifact lib for hadoop-code were in conflict with the expected jars by the CDH5.3.2 cluster we had.

3) Hadoop-core jars from Artifacts POM were excluded from the artifact package....
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<scope>provided</scope>
</dependency>

Things started to work as expected hereafter.

[#3]

Hi,

I applied point 1 with following syntax, and still not work:

$ hadoop jar JarName.jar ClassName -Doozie.launcher.mapreduce.task.classpath.user.precedence=true param1 param2

In my case, I have fat-jar with modified parquet lib in it. But cluster always picking default parquet lib in CDH. In standalone mode, my program is running fine.

[#4]

Using the class path precedence is not the correct solution for all cases.

A solution that will work in all cases is to use shading for the classes that you have modified versions of (use maven or gradle to do that).

In your case you need to shade the parquet classes that you have modified when you package the jar.

Be careful if you change classes like parquet: you could ed up with files that are only readable with your code and force you to keep packaging it with all jobs. That could cause problems later if you decide to use a different method to access the files.

Wilfred