Issues with YARN classpath after manual upgrade from 2.1 to 2.2

[#1]

I am performing a test manual upgrade of 2.1 to 2.2 using instructions at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/bk_upgrading_hdp_manually/content/ch_upgrade..., and am having issues on step number 8, which is found at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/bk_upgrading_hdp_manually/content/configure-...

When running the following (either with hadoop jar or yarn jar)

hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar randomwriter -Dtest.randomwrite.total_bytes=10000000 test-after-upgrade

I encounter

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService

…

I have tried a couple of different values for yarn.application.classpath in the yarn-site.xml, but no joy. I believe that this class is in /usr/hdp/2.2.0.0-2041/hadoop/client/hadoop-yarn-common.jar, which looks to be in the running config based on output from the tracking url web ui.

FROM yarn-site.xml (Note: I hardcoded some substitution variables to ensure that the substitution of hp.version and variable were not the problem, but there was no change)

<property>

<name>yarn.application.classpath</name>

<value>/etc/hadoop,

/etc/hadoop/conf,

/usr/hdp/2.2.0.0-2041/hadoop/client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*</value>

</property>

FROM CONFIG VALUES IN MR Tracking URL

<value>/etc/hadoop,

/etc/hadoop/conf,

/usr/hdp/2.2.0.0-2041/hadoop/client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,

/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*</value>

<source>yarn-site.xml</source>

------------------

Anyway, looks to me like /usr/hdp/2.2.0.0-2041/hadoop/client/* is in the classpath, which contains the hadoop-yarn-common.jar file, which in turn should contain this class.

Any obvious issues or pointers to drill down further? Thanks!

[#2]

I would contact support as this is not trivial upgrade.

[#3]

@jbarnett

export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS"

Do you have those entries in hadoop-env.sh and mapred-env.sh?

[#4]

Hi @jbarnett, Some of your paths are wrong. You cannot just replace "current" with "2.2.0.0-2041". Based on my directory structure in 2.2.8 all these paths are non-existent:

/usr/hdp/2.2.0.0-2041/hadoop-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-client/lib/*,
/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-hdfs-client/lib/*,
/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/*,
/usr/hdp/2.2.0.0-2041/hadoop-yarn-client/lib/*

[#5 · Accepted solution]

What a difference one character makes!!! The documentation has colons (like a normal classpath!) at the end of the mapreduce.application.classpath instead of commas. The following is the documented entry to put into mapred-site.xml.

<property> <name>mapreduce.application.classpath</name>
 <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*: 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/*: 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*: 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*: 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*: 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar: 
/etc/hadoop/conf/secure</value></property>

According to the Apache docs, this should be a “A comma-separated list of CLASSPATH entries.” So, change the colon to a comma, and we are good to go!

<property> <name>mapreduce.application.classpath</name> <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*, 
$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/*, 
$PWD/mr-framework/hadoop/share/hadoop/common/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/*, 
$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/*, 
$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*, 
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar, 
/etc/hadoop/conf/secure</value></property>

BTW, I ran with the HADOOP_OPTS as suggested, and then removed this and the job still ran.

[#6]

@jbarnett I notified the documentation team to fix this. Thanks for catching it. Just to confirm, you replaced colon as stated in the doc with , as in the apache docs and your problem was solved?

[#7]

@jeff there's an open Jira, just waiting for validation, internal bug ID BUG-53613

[#8]

@jbarnett - This has been fixed in our documentation, per internal BUG-53613. See http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_upgrading_hdp_manually/content/configure-... for a link to the 2.4.2 version. These changes were back ported all the way back to 2.2.8. Thanks for helping us to increase the quality of our documentation.