question Re: YARN service stops when started class not found exception in Archives of Support Questions (Read Only)

YARN service stops when started class not found exception

sim6 — Thu, 05 May 2016 14:29:28 GMT

In the logs, I get this classNotFoundException .NodeManager fails in a couple of seconds when started. Rest of the YARN components are working fine except for Nodemanager. YARN service check times out and never completes.

YARN logs:

2016-05-05 12:44:24,449 INFO event.AsyncDispatcher (AsyncDispatcher.java:register(208)) - Registering

2016-05-05 12:44:24,733 WARN  containermanager.AuxServices (AuxServices.java:serviceInit(130)) - The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.
2016-05-05 12:44:24,734 INFO  containermanager.AuxServices (AuxServices.java:addService(72)) - Adding auxiliary service httpshuffle, "mapreduce_shuffle"
2016-05-05 12:44:24,834 FATAL containermanager.AuxServices (AuxServices.java:serviceInit(145)) - Failed to initialize spark_shuffle
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:537)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:585)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)
	... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
	... 11 more

You can check the complete logs here:

http://warehouse.swtched.com:8088/logs/yarn-yarn-nodemanager-warehouse.swtched.com.log

Re: YARN service stops when started class not found exception

rajkumar_singh — Thu, 05 May 2016 14:30:31 GMT

its seems that you are running spark on same cluster and you have spark shuffle enable in your yarn-site.xml but it is not there in nodemanager classpath can you find spark shuffle jar at the location /usr/hdp/2.*/spark/lib/spark-*-yarn-shuffle.jar, add this jar to NM classpath and restart the NM

Re: YARN service stops when started class not found exception

nyadav — Thu, 05 May 2016 14:32:49 GMT

Hi Simarn,

Seems like you have defined "org.apache.spark.yarn.network.YarnShuffleService" in your yarn-site.xml, but the jar containing that class "spark-<version>-yarn-shuffle.jar" is missing in your nodemanager classpath. Please add it similar to mapreduce_shuffle class.

Re: YARN service stops when started class not found exception

aervits — Thu, 05 May 2016 14:35:05 GMT

please review Spark User Guide for the specific class you're missing http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_spark-guide/content/config-dra-manual.html

Re: YARN service stops when started class not found exception

sim6 — Thu, 05 May 2016 14:35:22 GMT

I do not have spark installed. In fact I thought that adding the service could possibly solve the issue? I believe spark thing is coming in the picture because it is in the list of auxiliary services. Shall I consider removing it from there?

Re: YARN service stops when started class not found exception

sim6 — Thu, 05 May 2016 14:36:48 GMT

@Artem Ervits: But should it not have been included automatically rather than I having to add it manually? I have installed YARN before and never really had to add it manually.

Re: YARN service stops when started class not found exception

aervits — Thu, 05 May 2016 14:41:11 GMT

consider removing references to Spark in your yarn-site.xml, in Ambari you'd go to configs section and use filter for spark option.

Re: YARN service stops when started class not found exception

jkumarmcse — Thu, 18 May 2017 02:25:52 GMT

yarn-site.xml - having entry like below

<name>yarn.nodemanager.aux-services.spark2_shuffle.classpath</name>

<value>/usr/hdp/${hdp.version}/spark2/aux/*</value>

</property>

Make sure you have the jar file ("spark-{version}-yarn-shuffle.jar" ) in that above mentioned location.

If not we can create a create directory and copy that file from any other location.

Now it is working fine.

Re: YARN service stops when started class not found exception

dattaningole — Wed, 27 Dec 2017 18:04:10 GMT

I got stucked with same issue. I just copy the spark-2.1.1.2.6.1.0-129-yarn-shuffle.jar in /usr/hdp/2.6.1.0-129/spark/aux location. It is working fine

Re: YARN service stops when started class not found exception

giri_pawan0206 — Thu, 07 Feb 2019 20:18:06 GMT

hi All,

I would like share that this has been fixed by changing yarn-site.xml file.

OLD Parameters -

<code><property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle,spark_shuffle,spark2_shuffle</value>
</property>

New Parameters

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

Please Note - this is for those users who has install Yarn+MRV2 and not for spark & spark 2.

Thanks
Pawan Giri