Reply
New Contributor
Posts: 3
Registered: ‎06-30-2016
Accepted Solution

YARN Nodemanager Restart Fails with Spark 2 Beta installed

Nodemanagers fail to restart after adding Spark 2 Beta service. We have added the gateway role to all nodes and have the spark1 service installed. YARN ResourceManager and YARN History Server start fine but nodemanagers fail to start with the following error.

 

Role failed to start due to error com.cloudera.cmf.service.config.ConfigGenException: Conflicting yarn extensions of key [spark_shuffle]. 

Cloudera Employee
Posts: 3
Registered: ‎10-19-2014

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

Hey astadtler,

 

Which version of CDH are you running?

New Contributor
Posts: 3
Registered: ‎06-30-2016

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

CDH 5.7.1
CM 5.7.1

Looks like this error is related to Spark 2.0 Beta requiring CM 5.8.
Highlighted
Cloudera Employee
Posts: 3
Registered: ‎10-19-2014

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

Hey astadtler,

 

The Spark 2.0 Beta requires CM 5.8. Please install CM5.8 on your CDH5.7.1 cluster. Installing CM is a lightweight process. This error is likely because you are using CM5.7. 

New Contributor
Posts: 1
Registered: ‎02-03-2016

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

[ Edited ]

Hi, 

 

I have the same problem with Cloudera 5.11

 

No se ha podido iniciar el rol debido al error com.cloudera.cmf.service.config.ConfigGenException: Conflicting yarn extensions provided by more than one service. Yarn extension key [spark_shuffle], First value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=false, spark.shuffle.service.port=7337}, service=Spark 2}], Second value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=true, spark.shuffle.service.port=7338}, service=Spark}].

Posts: 588
Kudos: 66
Solutions: 34
Registered: ‎04-06-2015

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed


innovanngg wrote:

Hi, 

 

I have the same problem with Cloudera 5.11

 


If you are having issues with Spark 2 rather than the Spark 2 beta, I would suggest starting a new thread on the Advanced Analytics (Apache Spark) board. Make sure to include your setup information as well as the steps taken that led to the error. 

 




Cy Jervis, Community Manager - I'm not an expert but will supply relevant content from time to time. :)

Learn more about the Cloudera Community:


Terms of Service


Community Guidelines


How to use the forum

Announcements

Currently incubating in Cloudera Labs:

Envelope
HTrace
Ibis
Impyla
Livy
Oryx
Phoenix
Spark Runner for Beam SDK
Time Series for Spark
YCSB