YARN service stops when started class not found exception

[#1]

In the logs, I get this classNotFoundException .NodeManager fails in a couple of seconds when started. Rest of the YARN components are working fine except for Nodemanager. YARN service check times out and never completes.

YARN logs:

2016-05-05 12:44:24,449 INFO event.AsyncDispatcher (AsyncDispatcher.java:register(208)) - Registering

2016-05-05 12:44:24,733 WARN  containermanager.AuxServices (AuxServices.java:serviceInit(130)) - The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.
2016-05-05 12:44:24,734 INFO  containermanager.AuxServices (AuxServices.java:addService(72)) - Adding auxiliary service httpshuffle, "mapreduce_shuffle"
2016-05-05 12:44:24,834 FATAL containermanager.AuxServices (AuxServices.java:serviceInit(145)) - Failed to initialize spark_shuffle
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:537)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:585)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)
	... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
	... 11 more

You can check the complete logs here:

http://warehouse.swtched.com:8088/logs/yarn-yarn-nodemanager-warehouse.swtched.com.log

[#2]

its seems that you are running spark on same cluster and you have spark shuffle enable in your yarn-site.xml but it is not there in nodemanager classpath can you find spark shuffle jar at the location /usr/hdp/2.*/spark/lib/spark-*-yarn-shuffle.jar, add this jar to NM classpath and restart the NM

[#3]

I do not have spark installed. In fact I thought that adding the service could possibly solve the issue? I believe spark thing is coming in the picture because it is in the list of auxiliary services. Shall I consider removing it from there?

[#4]

Hi Simarn,

Seems like you have defined "org.apache.spark.yarn.network.YarnShuffleService" in your yarn-site.xml, but the jar containing that class "spark-<version>-yarn-shuffle.jar" is missing in your nodemanager classpath. Please add it similar to mapreduce_shuffle class.

[#5 · Accepted solution]

please review Spark User Guide for the specific class you're missing http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_spark-guide/content/config-dra-manual.htm...

[#6]

@Artem Ervits: But should it not have been included automatically rather than I having to add it manually? I have installed YARN before and never really had to add it manually.

[#7]

consider removing references to Spark in your yarn-site.xml, in Ambari you'd go to configs section and use filter for spark option.

[#8]

yarn-site.xml - having entry like below

<property>

<name>yarn.nodemanager.aux-services.spark2_shuffle.classpath</name>

<value>/usr/hdp/${hdp.version}/spark2/aux/*</value>

</property>

Make sure you have the jar file ("spark-{version}-yarn-shuffle.jar" ) in that above mentioned location.

If not we can create a create directory and copy that file from any other location.

Now it is working fine.

[#9]

I got stucked with same issue. I just copy the spark-2.1.1.2.6.1.0-129-yarn-shuffle.jar in /usr/hdp/2.6.1.0-129/spark/aux location. It is working fine

[#10]

hi All,

I would like share that this has been fixed by changing yarn-site.xml file.

OLD Parameters -

<code><property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle,spark_shuffle,spark2_shuffle</value>
</property>

New Parameters

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

Please Note - this is for those users who has install Yarn+MRV2 and not for spark & spark 2.

Thanks
Pawan Giri