Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Solved Go to solution
Highlighted

Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Expert Contributor

I am performing a test manual upgrade of 2.1 to 2.2 using instructions at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/bk_upgrading_hdp_manually/content/ch_upgrade..., and am having issues on step number 8, which is found at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/bk_upgrading_hdp_manually/content/configure-...

When running the following (either with hadoop jar or yarn jar)

hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar randomwriter -Dtest.randomwrite.total_bytes=10000000 test-after-upgrade

I encounter

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService

I have tried a couple of different values for yarn.application.classpath in the yarn-site.xml, but no joy. I believe that this class is in /usr/hdp/2.2.0.0-2041/hadoop/client/hadoop-yarn-common.jar, which looks to be in the running config based on output from the tracking url web ui.

FROM yarn-site.xml (Note: I hardcoded some substitution variables to ensure that the substitution of hp.version and variable were not the problem, but there was no change)

<property>

<name>yarn.application.classpath</name>

<value>/etc/hadoop,

/etc/hadoop/conf,

/usr/hdp/2.2.0.0-2041/hadoop/client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*</value>

</property>

FROM CONFIG VALUES IN MR Tracking URL

<value>/etc/hadoop,

/etc/hadoop/conf,

/usr/hdp/2.2.0.0-2041/hadoop/client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*</value>

<source>yarn-site.xml</source>

------------------

Anyway, looks to me like /usr/hdp/2.2.0.0-2041/hadoop/client/* is in the classpath, which contains the hadoop-yarn-common.jar file, which in turn should contain this class.

Any obvious issues or pointers to drill down further? Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Expert Contributor

What a difference one character makes!!! The documentation has colons (like a normal classpath!) at the end of the mapreduce.application.classpath instead of commas. The following is the documented entry to put into mapred-site.xml.

<property> <name>mapreduce.application.classpath</name>
 <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*: 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*: 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar: 
/etc/hadoop/conf/secure</value></property>

According to the Apache docs, this should be a “A comma-separated list of CLASSPATH entries.” So, change the colon to a comma, and we are good to go!

<property> <name>mapreduce.application.classpath</name> <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*, 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*, 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar, 
/etc/hadoop/conf/secure</value></property>

BTW, I ran with the HADOOP_OPTS as suggested, and then removed this and the job still ran.

View solution in original post

7 REPLIES 7
Highlighted

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Mentor

I would contact support as this is not trivial upgrade.

Highlighted

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

@jbarnett

export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS"

Do you have those entries in hadoop-env.sh and mapred-env.sh?

Highlighted

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Hi @jbarnett, Some of your paths are wrong. You cannot just replace "current" with "2.2.0.0-2041". Based on my directory structure in 2.2.8 all these paths are non-existent:

/usr/hdp/2.2.0.0-2041/hadoop-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,
/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,
/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*
Highlighted

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Expert Contributor

What a difference one character makes!!! The documentation has colons (like a normal classpath!) at the end of the mapreduce.application.classpath instead of commas. The following is the documented entry to put into mapred-site.xml.

<property> <name>mapreduce.application.classpath</name>
 <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*: 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*: 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar: 
/etc/hadoop/conf/secure</value></property>

According to the Apache docs, this should be a “A comma-separated list of CLASSPATH entries.” So, change the colon to a comma, and we are good to go!

<property> <name>mapreduce.application.classpath</name> <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*, 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*, 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar, 
/etc/hadoop/conf/secure</value></property>

BTW, I ran with the HADOOP_OPTS as suggested, and then removed this and the job still ran.

View solution in original post

Highlighted

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Mentor

@jbarnett I notified the documentation team to fix this. Thanks for catching it. Just to confirm, you replaced colon as stated in the doc with , as in the apache docs and your problem was solved?

Highlighted

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Mentor

@jeff there's an open Jira, just waiting for validation, internal bug ID BUG-53613

Highlighted

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Explorer

@jbarnett - This has been fixed in our documentation, per internal BUG-53613. See http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_upgrading_hdp_manually/content/configure-... for a link to the 2.4.2 version. These changes were back ported all the way back to 2.2.8. Thanks for helping us to increase the quality of our documentation.

Don't have an account?
Coming from Hortonworks? Activate your account here