Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark_shuffle fails nodemanager

avatar
New Contributor

 

 Hi,

 

i have cds 2.3.2 on cdh 5.15.

  

i have configured shuffle as under,-

 

<property><name>yarn.nodemanager.aux-services</name><value>spark_shuffle</value></property><property><name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name><value>org.apache.spark.network.yarn.YarnShuffleService</value></property><property><name>yarn.resourcemanager.hostname</name><value>whf00aql.in.oracle.com</value></property><property><name>spark.yarn.shuffle.stopOnFailure</name><value>false</value></property>

 

Also distributed the shuffle jar as exected for external shuffle config.

 

on restart of yarn i see teh following,-

 

 

Role failed to start due to error com.cloudera.cmf.service.config.ConfigGenException:
Conflicting yarn extensions provided by more than one service. Yarn extension key [spark_shuffle], 
First value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=true, spark.shuffle.service.port=7337}, service=Spark 2}], Second value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=false, spark.shuffle.service.port=7337}, service=Spark}].

 

please help

1 ACCEPTED SOLUTION

avatar
New Contributor
Issue was with spark authentication set up between spark and spar2 in cdh. default value of spark2 did not match to spark. once changed it resolved the issue spark.authenticate=true

View solution in original post

1 REPLY 1

avatar
New Contributor
Issue was with spark authentication set up between spark and spar2 in cdh. default value of spark2 did not match to spark. once changed it resolved the issue spark.authenticate=true