Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

Highlighted

Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

Hi,

We are currently upgrading from HDP 2.3.2 to HDP 2.3.4 using express upgrade but it is failing in restarting YARN node manager.

In the logs, we found this error:

2016-05-04 04:31:08,517 FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(540)) - Error starting NodeManager

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:537)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:585)

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)

... 10 more

Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)

... 11 more

2016-05-04 04:31:08,520 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NodeManager at hadoop-master01.solutionworks.com/10.1.225.80

************************************************************/

11 REPLIES 11
Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

@AICS Basis

Are you using spark also ?

Can you make sure respective jars [spark-<version>-yarn-shuffle.jar] for spark are available in classpath on nodemanagers?

Pls check in the link for below statement resolution- https://scn.sap.com/blogs/vora/2015/12/09/sap-hana-vora--troubleshooting

[[15] Yarn NodeManager with Spark Controller fails with java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService]

Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

I think you have spark shuffle enable in your yarn-site.xml but it is not there in nodemanager classpath can you find spark shuffle jar at the location /usr/hdp/2.4.0.0-169/spark/lib/spark-1.6.0.2.4.0.0-169-yarn-shuffle.jar, add this jar to NM classpath and restart the NM

Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

Hi Rajkumar,

In editing namenode classpath, do we edit the yarn-site.xml and update this property?

<property> <name>yarn.application.classpath</name> <value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-c lient/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/* ,/usr/hdp/2.3.4.0-3485/spark/lib/*</value> </property>

Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

Hi,

Yes, we have also spark installed. We have also added the spark shuffle jar in our nodemanager classpath via YARN configuration in ambari. Here is the content

$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/* ,/usr/hdp/2.3.4.0-3485/spark/lib/*.

However, in trying to restart the nodemanager via Ambari it is getting stuck up, is it because we are in the middle of the upgrade?

Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

@AICS Basis Can you copy the jar on all NM in the path mentioned in the link - https://scn.sap.com/blogs/vora/2015/12/09/sap-hana-vora--troubleshooting

Also do check the permissions and give a re-try

Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

Hi Sagar,

Yes, this is to confirm that the jar file location is defined in the class path of all NM in our cluster. However, nodemanager cannot be started.

Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

@AICS Basis

Here is the quick fix, I just faced this issue.

Don't put any extra confs or parameters. Just copy the spark-<version>-yarn-shuffle.jar into /usr/hdp/current/hadoop-yarn-nodemanager/lib/ direcotry on node where node manager causing issue. It is strange that spark jar is needed for node manager to start, so I will share this finding to our Dev team.

Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

*WORKED* - NodeManager Only. Still HiveServer2 is DOWN. Working on it. May be some thing to do with NameNode - going down intermittently or timing out copying big usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz

For NodeManager

1: Got yarn-shuffle.jar from HDP2.4 Sandbox and Uploaded to CLuster-Node

2: cp ~/spark-1.6.2.2.4.3.0-227-yarn-shuffle.jar /usr/hdp/2.4.3.0-227/spark/lib/

3: cp /usr/hdp/2.4.3.0-227/spark/lib/spark-1.6.2.2.4.3.0-227-yarn-shuffle.jar /usr/hdp/current/hadoop-yarn-nodemanager/lib/

4: Add: vi /usr/hdp/2.4.3.0-227/etc/spark/conf/spark-defaults.conf

spark.dynamicAllocation.enabled true

spark.shuffle.service.enabled true

5: Restart Cluster

Highlighted

Re: Express Upgrade to HDP 2.3.4 failed on restarting YARN Node managers

@Todd Wilson @Artem Ervits

*Atlas"

1: manually

cd /usr/hdp/current/atlas-server/bin

./atlas_start.py

2: Then go to ambari and start "Atlas Metadata Server"

3: Works for a while - but goes down again.

Since this is Single Node Cluster - how can I remove Atlas Client - that I accidentally installed? Just the Client. Will keep the service on.

Log: cd /var/log/atlas/

Complain on Missing atlas-client.properties. I dont have that file in "atlas-client/conf" dir. in Sandbox - we dont have "atlas-client" folder too.

Don't have an account?
Coming from Hortonworks? Activate your account here