Created on 08-02-2017 07:08 AM - edited 09-16-2022 05:02 AM
After add
<property> <name>yarn.nodemanager.aux-services</name> <value>spark_shuffle,mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.spark_shuffle.class</name> <value>org.apache.spark.network.yarn.YarnShuffleService</value> </property>
I am getting the following error:
2017-08-02 13:52:06,207 FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize spark_shuffle java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2234) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:236) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:318) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2202) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2226) ... 10 more Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2108) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2200)
I have in my CLASSPATH the library need it
<property> <name>yarn.application.classpath</name> <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,/opt/spark/yarn/spark-2.1.0-yarn-shuffle.jar</value> </property> <property>
Any ideas?
Created 08-02-2017 03:13 PM
Created 08-02-2017 11:55 PM
Created 08-07-2017 02:58 AM
After adding the "classpath" when I execute,
hadoop classpath, it doesn't return the path that I have added.
Any ideas?