Cloudera Labs
Provide feedback on Cloudera Labs
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

YARN Nodemanager Restart Fails with Spark 2 Beta installed

Solved Go to solution
Highlighted

YARN Nodemanager Restart Fails with Spark 2 Beta installed

New Contributor

Nodemanagers fail to restart after adding Spark 2 Beta service. We have added the gateway role to all nodes and have the spark1 service installed. YARN ResourceManager and YARN History Server start fine but nodemanagers fail to start with the following error.

 

Role failed to start due to error com.cloudera.cmf.service.config.ConfigGenException: Conflicting yarn extensions of key [spark_shuffle]. 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

Cloudera Employee

Hey astadtler,

 

The Spark 2.0 Beta requires CM 5.8. Please install CM5.8 on your CDH5.7.1 cluster. Installing CM is a lightweight process. This error is likely because you are using CM5.7. 

View solution in original post

6 REPLIES 6
Highlighted

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

Cloudera Employee

Hey astadtler,

 

Which version of CDH are you running?

Highlighted

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

New Contributor
CDH 5.7.1
CM 5.7.1

Looks like this error is related to Spark 2.0 Beta requiring CM 5.8.
Highlighted

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

Cloudera Employee

Hey astadtler,

 

The Spark 2.0 Beta requires CM 5.8. Please install CM5.8 on your CDH5.7.1 cluster. Installing CM is a lightweight process. This error is likely because you are using CM5.7. 

View solution in original post

Highlighted

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

New Contributor

Hi, 

 

I have the same problem with Cloudera 5.11

 

No se ha podido iniciar el rol debido al error com.cloudera.cmf.service.config.ConfigGenException: Conflicting yarn extensions provided by more than one service. Yarn extension key [spark_shuffle], First value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=false, spark.shuffle.service.port=7337}, service=Spark 2}], Second value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=true, spark.shuffle.service.port=7338}, service=Spark}].

Highlighted

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

Community Manager

@innovanngg wrote:

Hi, 

 

I have the same problem with Cloudera 5.11

 


If you are having issues with Spark 2 rather than the Spark 2 beta, I would suggest starting a new thread on the Advanced Analytics (Apache Spark) board. Make sure to include your setup information as well as the steps taken that led to the error. 


Cy Jervis, Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:
Community Guidelines
How to use the forum

Re: YARN Nodemanager Restart Fails with Spark 2 Beta installed

I had the similar issue with Spark2 CSD based installation in a secured cluster.

 

But later it is rectified when i enabled Spark authentication.

Don't have an account?
Coming from Hortonworks? Activate your account here