Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎11-13-2018
Accepted Solution

spark_shuffle fails nodemanager

[ Edited ]

 

 Hi,

 

i have cds 2.3.2 on cdh 5.15.

  

i have configured shuffle as under,-

 

<property><name>yarn.nodemanager.aux-services</name><value>spark_shuffle</value></property><property><name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name><value>org.apache.spark.network.yarn.YarnShuffleService</value></property><property><name>yarn.resourcemanager.hostname</name><value>whf00aql.in.oracle.com</value></property><property><name>spark.yarn.shuffle.stopOnFailure</name><value>false</value></property>

 

Also distributed the shuffle jar as exected for external shuffle config.

 

on restart of yarn i see teh following,-

 

 

Role failed to start due to error com.cloudera.cmf.service.config.ConfigGenException:
Conflicting yarn extensions provided by more than one service. Yarn extension key [spark_shuffle], 
First value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=true, spark.shuffle.service.port=7337}, service=Spark 2}], Second value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=false, spark.shuffle.service.port=7337}, service=Spark}].

 

please help

New Contributor
Posts: 3
Registered: ‎11-13-2018

Re: spark_shuffle fails nodemanager

Issue was with spark authentication set up between spark and spar2 in cdh. default value of spark2 did not match to spark. once changed it resolved the issue spark.authenticate=true
Announcements