Support Questions
Find answers, ask questions, and share your expertise

Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Solved Go to solution

Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Expert Contributor

I am performing a test manual upgrade of 2.1 to 2.2 using instructions at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/bk_upgrading_hdp_manually/content/ch_upgrade..., and am having issues on step number 8, which is found at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/bk_upgrading_hdp_manually/content/configure-...

When running the following (either with hadoop jar or yarn jar)

hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar randomwriter -Dtest.randomwrite.total_bytes=10000000 test-after-upgrade

I encounter

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService

I have tried a couple of different values for yarn.application.classpath in the yarn-site.xml, but no joy. I believe that this class is in /usr/hdp/2.2.0.0-2041/hadoop/client/hadoop-yarn-common.jar, which looks to be in the running config based on output from the tracking url web ui.

FROM yarn-site.xml (Note: I hardcoded some substitution variables to ensure that the substitution of hp.version and variable were not the problem, but there was no change)

<property>

<name>yarn.application.classpath</name>

<value>/etc/hadoop,

/etc/hadoop/conf,

/usr/hdp/2.2.0.0-2041/hadoop/client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*</value>

</property>

FROM CONFIG VALUES IN MR Tracking URL

<value>/etc/hadoop,

/etc/hadoop/conf,

/usr/hdp/2.2.0.0-2041/hadoop/client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*</value>

<source>yarn-site.xml</source>

------------------

Anyway, looks to me like /usr/hdp/2.2.0.0-2041/hadoop/client/* is in the classpath, which contains the hadoop-yarn-common.jar file, which in turn should contain this class.

Any obvious issues or pointers to drill down further? Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Expert Contributor

What a difference one character makes!!! The documentation has colons (like a normal classpath!) at the end of the mapreduce.application.classpath instead of commas. The following is the documented entry to put into mapred-site.xml.

<property> <name>mapreduce.application.classpath</name>
 <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*: 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*: 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar: 
/etc/hadoop/conf/secure</value></property>

According to the Apache docs, this should be a “A comma-separated list of CLASSPATH entries.” So, change the colon to a comma, and we are good to go!

<property> <name>mapreduce.application.classpath</name> <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*, 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*, 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar, 
/etc/hadoop/conf/secure</value></property>

BTW, I ran with the HADOOP_OPTS as suggested, and then removed this and the job still ran.

View solution in original post

7 REPLIES 7

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Mentor

I would contact support as this is not trivial upgrade.

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

@jbarnett

export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS"

Do you have those entries in hadoop-env.sh and mapred-env.sh?

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Hi @jbarnett, Some of your paths are wrong. You cannot just replace "current" with "2.2.0.0-2041". Based on my directory structure in 2.2.8 all these paths are non-existent:

/usr/hdp/2.2.0.0-2041/hadoop-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,
/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,
/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Expert Contributor

What a difference one character makes!!! The documentation has colons (like a normal classpath!) at the end of the mapreduce.application.classpath instead of commas. The following is the documented entry to put into mapred-site.xml.

<property> <name>mapreduce.application.classpath</name>
 <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*: 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*: 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar: 
/etc/hadoop/conf/secure</value></property>

According to the Apache docs, this should be a “A comma-separated list of CLASSPATH entries.” So, change the colon to a comma, and we are good to go!

<property> <name>mapreduce.application.classpath</name> <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*, 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*, 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar, 
/etc/hadoop/conf/secure</value></property>

BTW, I ran with the HADOOP_OPTS as suggested, and then removed this and the job still ran.

View solution in original post

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Mentor

@jbarnett I notified the documentation team to fix this. Thanks for catching it. Just to confirm, you replaced colon as stated in the doc with , as in the apache docs and your problem was solved?

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Mentor

@jeff there's an open Jira, just waiting for validation, internal bug ID BUG-53613

Re: Issues with YARN classpath after manual upgrade from 2.1 to 2.2

Explorer

@jbarnett - This has been fixed in our documentation, per internal BUG-53613. See http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_upgrading_hdp_manually/content/configure-... for a link to the 2.4.2 version. These changes were back ported all the way back to 2.2.8. Thanks for helping us to increase the quality of our documentation.