We are currently working on testing some of our older workflows from hadoop 4.x in 6.x (yes large jump)
and we have a particular phase of a workflow fail which seems to have to do with the classpath. We now in our 4.x cluster that the classpath for yarn in 4.x shows as:
4.x yarn classpath:
/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hbase/hbase.jar:/usr/lib/hbase/hbase-0.94.6-cdh4.3.0-security-tests.jar:/usr/lib/hbase/conf:/usr/lib/hbase/lib/zookeeper.jar::/contrib/capacity-scheduler/*.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/*:/usr/lib/hadoop-0.20-mapreduce/.//*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*
yet in 6.3.2 it shows as:
6.3.2 classpath for yarn
/etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-yarn/.//*
so when our workflow is ran it 6.x it fails with:
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
But, if I manually the phase that's failing
java -cp my-jar.jar:`hbase classpath` MyMain my args
It works fine. If instead of hbase classpath I use hadoop classpath, I get the exact same error:
java -cp my-jar.jar:`hadoop classpath` MyMain my args
...
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
So, we know it's the classpath. But we haven't found a way for oozie/yarn to send us the right classpath.
We have tried updating the classpath.txt file as a test and it still fails. We have also tried updating the classpath for yarn and yet again fails. So where in the world do we add our classpath in 6.x that is needed?
NOTE; We don't bundle any of our jars so we knows its not that.
I know I've seen articles about having to add this to HADOOP_OPTS, but we don't know where that is in cdh 6.x. Can someone please point us in the right direction or give us a helping hand? I can post logs and debug info if needed (thus the reason we are trying this on 6.3.x 🙂 🙂 ..
