Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.5 Node manager failing to start due to ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

HDP 2.5 Node manager failing to start due to ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

Explorer

Installed HDP 2.5 using Ambari 2.4.x. Node managers in all nodes are failing to start due to ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService. Based on thread [1] attempted the fix but spark-*yarn-shuffle.jar is missing in the installation. Is there a work around ?

[1] - https://community.hortonworks.com/questions/44703/node-manager-is-failing-again-and-again-on-one-of....

4 REPLIES 4
Highlighted

Re: HDP 2.5 Node manager failing to start due to ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

Explorer

Node managers are starting after changing the value for yarn.nodemanager.aux-services in yarn configuration to

only mapreduce_shuffle, removing spark_shuffle and spark2_shuffle. Just wondering what is the impact of this change ?

Highlighted

Re: HDP 2.5 Node manager failing to start due to ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

Super Guru
Highlighted

Re: HDP 2.5 Node manager failing to start due to ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

Explorer

I have seen this thread based on which I made the changes in yarn configuration. My question was that why the spark-*yarn-shuffle.jar file missing in 2.5 installation ?

Highlighted

Re: HDP 2.5 Node manager failing to start due to ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

Super Collaborator

@Vinuraj M

In HDP2.5/Ambari2.4 we have properties set for yarn.nodemanager.aux-services.spark_shuffle.* and yarn.nodemanager.aux-services.spark2_shuffle. Which are set to {{stack_root}}/${hdp.version}/spark*/aux/* .

spark yarn shuffle jar are installed while HDP2.5 installation, if jars are missing you can install in using yum command.

#yum install spark2_2_5_0_0_1245-yarn-shuffle

#yum install spark_2_5_0_0_1245-yarn-shuffle

Don't have an account?
Coming from Hortonworks? Activate your account here