Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBASE_HOME path no longer in YARN classpath?

HBASE_HOME path no longer in YARN classpath?

Explorer

We are currently working on testing some of our older workflows from hadoop 4.x in 6.x (yes large jump)

and we have a particular phase of a workflow fail which seems to have to do with the classpath. We now in our 4.x cluster that the classpath for yarn in 4.x shows as:

 

 

4.x yarn classpath:
/etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hbase/hbase.jar:/usr/lib/hbase/hbase-0.94.6-cdh4.3.0-security-tests.jar:/usr/lib/hbase/conf:/usr/lib/hbase/lib/zookeeper.jar::/contrib/capacity-scheduler/*.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/*:/usr/lib/hadoop-0.20-mapreduce/.//*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-yarn/lib/*

 

yet in 6.3.2 it shows as:

 

6.3.2 classpath for yarn
/etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-yarn/.//*

 

 so when our workflow is ran it 6.x it fails with:

 

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes

 

But,  if I manually the phase that's failing

java -cp my-jar.jar:`hbase classpath` MyMain my args
 
It works fine.  If instead of hbase classpath I use hadoop classpath, I get the exact same error:
 
java -cp my-jar.jar:`hadoop classpath` MyMain my args
...
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
So, we know it's the classpath.  But we haven't found a way for oozie/yarn to send us the right classpath.
 
We have tried updating the classpath.txt file as a test and it still fails.  We have also tried updating the classpath for yarn and yet again fails.  So where in the world do we add our classpath in 6.x that is needed?
 
NOTE;  We don't bundle any of our jars so we knows its not that.  
 
I know I've seen articles about having to add this to HADOOP_OPTS, but we don't know where that is in cdh 6.x.  Can someone please point us in the right direction or give us a helping hand?  I can post logs and debug info if needed (thus the reason we are trying this on 6.3.x ..  
 

 

1 REPLY 1
Highlighted

Re: HBASE_HOME path no longer in YARN classpath?

Explorer

In 6.3.x are we needing to set something in  oozie.service.HadoopAccessorService.hadoop.configurations   

just trying to get to the bottom of this.  

 

Don't have an account?
Coming from Hortonworks? Activate your account here