Created 03-11-2020 08:03 AM
We are currently working on testing some of our older workflows from hadoop 4.x in 6.x (yes large jump)
and we have a particular phase of a workflow fail which seems to have to do with the classpath. We now in our 4.x cluster that the classpath for yarn in 4.x shows as:
4.x yarn classpath:
/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hbase/hbase.jar:/usr/lib/hbase/hbase-0.94.6-cdh4.3.0-security-tests.jar:/usr/lib/hbase/conf:/usr/lib/hbase/lib/zookeeper.jar::/contrib/capacity-scheduler/*.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/*:/usr/lib/hadoop-0.20-mapreduce/.//*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*
yet in 6.3.2 it shows as:
6.3.2 classpath for yarn
/etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-yarn/.//*
so when our workflow is ran it 6.x it fails with:
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
But, if I manually the phase that's failing
java -cp my-jar.jar:`hbase classpath` MyMain my args
java -cp my-jar.jar:`hadoop classpath` MyMain my args...
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
Created 03-11-2020 10:22 PM
In 6.3.x are we needing to set something in oozie.service.HadoopAccessorService.hadoop.configurations
just trying to get to the bottom of this.