Reply
New Contributor
Posts: 3
Registered: ‎06-30-2016
Accepted Solution

YARN Nodemanager Restart Fails with Spark 2 Beta installed

Nodemanagers fail to restart after adding Spark 2 Beta service. We have added the gateway role to all nodes and have the spark1 service installed. YARN ResourceManager and YARN History Server start fine but nodemanagers fail to start with the following error.

 

Role failed to start due to error com.cloudera.cmf.service.config.ConfigGenException: Conflicting yarn extensions of key [spark_shuffle]. 

Cloudera Employee
Posts: 3
Registered: ‎10-19-2014

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

Hey astadtler,

 

Which version of CDH are you running?

New Contributor
Posts: 3
Registered: ‎06-30-2016

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

CDH 5.7.1
CM 5.7.1

Looks like this error is related to Spark 2.0 Beta requiring CM 5.8.
Cloudera Employee
Posts: 3
Registered: ‎10-19-2014

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

Hey astadtler,

 

The Spark 2.0 Beta requires CM 5.8. Please install CM5.8 on your CDH5.7.1 cluster. Installing CM is a lightweight process. This error is likely because you are using CM5.7. 

New Contributor
Posts: 1
Registered: ‎02-03-2016

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

[ Edited ]

Hi, 

 

I have the same problem with Cloudera 5.11

 

No se ha podido iniciar el rol debido al error com.cloudera.cmf.service.config.ConfigGenException: Conflicting yarn extensions provided by more than one service. Yarn extension key [spark_shuffle], First value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=false, spark.shuffle.service.port=7337}, service=Spark 2}], Second value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=true, spark.shuffle.service.port=7338}, service=Spark}].

Posts: 742
Kudos: 89
Solutions: 46
Registered: ‎04-06-2015

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed


innovanngg wrote:

Hi, 

 

I have the same problem with Cloudera 5.11

 


If you are having issues with Spark 2 rather than the Spark 2 beta, I would suggest starting a new thread on the Advanced Analytics (Apache Spark) board. Make sure to include your setup information as well as the steps taken that led to the error. 

Cy Jervis, Community Manager


Was your question answered? Make sure to mark it as an accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Highlighted
Explorer
Posts: 10
Registered: ‎03-29-2017

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

I had the similar issue with Spark2 CSD based installation in a secured cluster.

 

But later it is rectified when i enabled Spark authentication.

Announcements

Currently incubating in Cloudera Labs:

Envelope
HTrace
Ibis
Impyla
Livy
Oryx
Phoenix
Spark Runner for Beam SDK
Time Series for Spark
YCSB