Thank you for the additional details! at org.apache.hadoop.util.RunJar.main(RunJar.java:212) This indicates a problem in the driver-end, or as you say 'during the execution of the job controller'. The issue is that even if you do add something to the MR distributed cache classpath, your executor class also references the same class. The act of adding a jar to the distributed task's classpath does not also add it to the local one. Here's how you can ensure that, if you use 'hadoop jar' to execute your job: ~> export HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hive/lib/hive-exec.jar
~> hadoop jar your-app.jar your.main.Class [arguments] This will add it also to your local JVM classpath, while your code will further add it onto the remote execution classpaths. > Optimally, I shouldn't have to stuff this one in the distributed cache since it sits in /opt/cloudera/parcels/CDH-5.3.5-1.cdh5.3.5.p0.4/jars/hive-exec-0.13.1-cdh5.3.5.jar on all of my slave nodes, but I also can't figure out how to tell MapReduce to look there. MR remote execution classpath is governed by the classpath entry defined in the mapred-site.xml and yarn-site.xml, and the additonal elements you add to the DistributedCache. They do not use the entire /opt/cloudera/parcels/CDH/jars/* path - this is so for isolation and flexibility purposes, as that area may carry multiple versions of the same dependencies, etc. Does this help?
... View more
