newly added Node, Yarn NodeManager not starting up - java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

[#1]

HI All -

i've HDP 2.5, and have a newly added Node to my cluster.

The node Manager is not starting up, the error is as shown below ->

Any ideas why this would be happening, & what needs to be done to fix ?

-----------------------------------

org.apache.hadoop.service.ServiceStateException: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:546)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)Caused by: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleServiceat java.net.URLClassLoader.findClass(URLClassLoader.java:381)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)at java.lang.ClassLoader.loadClass(ClassLoader.java:357)at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197)at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:348)at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:131)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)... 8 more

[#2 · Accepted solution]

spark2 service was not installed on the new node, and the nodemanager startup was expecting the following jar /usr/hdp/2.5.3.0-37/spark2/aux/spark-2.0.0.2.5.3.0-37-yarn-shuffle.jar, which was not there.

I re-installed spark2 service, and the issue is fixed.