Support Questions

Find answers, ask questions, and share your expertise

Namenodes don't start after add spark_suffle to yarn-site.xml

avatar
Explorer

 

After add 

 

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>spark_shuffle,mapreduce_shuffle</value>
  </property>
<property>
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
    <value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>

 

I am getting the following error:

 

 

2017-08-02 13:52:06,207 FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize spark_shuffle
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2234)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:236)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:318)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2202)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2226)
	... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2108)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2200)

 

I have in my CLASSPATH the library need it

 

<property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,/opt/spark/yarn/spark-2.1.0-yarn-shuffle.jar</value>
  </property>
  <property>

Any ideas?

 

3 REPLIES 3

avatar
Champion
Run 'jar -tf /opt/spark/yarn/spark-2.1.0-yarn-shuffle.jar | grep -i YarnShuffleService'

This will tell you if the jar file contains the class with the correct name.

avatar
Explorer
good spot! This is the output: jar -tf /opt/spark/yarn/spark-2.1.0-yarn-shuffle.jar | grep -i 'YarnShuffleService' org/apache/spark/network/yarn/YarnShuffleService$AppId.class org/apache/spark/network/yarn/YarnShuffleService.class

avatar
Explorer

After adding the "classpath" when I execute, 

 

hadoop classpath, it doesn't return the path that I have added.

 

Any ideas?