spark_shuffle fails nodemanager

Rayp — Fri, 16 Sep 2022 13:53:32 GMT

Hi,

i have cds 2.3.2 on cdh 5.15.

i have configured shuffle as under,-

<property><name>yarn.nodemanager.aux-services</name><value>spark_shuffle</value></property><property><name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name><value>org.apache.spark.network.yarn.YarnShuffleService</value></property><property><name>yarn.resourcemanager.hostname</name><value>whf00aql.in.oracle.com</value></property><property><name>spark.yarn.shuffle.stopOnFailure</name><value>false</value></property>

Also distributed the shuffle jar as exected for external shuffle config.

on restart of yarn i see teh following,-

Role failed to start due to error com.cloudera.cmf.service.config.ConfigGenException:
Conflicting yarn extensions provided by more than one service. Yarn extension key [spark_shuffle], 
First value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=true, spark.shuffle.service.port=7337}, service=Spark 2}], Second value [YarnAuxServiceExtension{className='org.apache.spark.network.yarn.YarnShuffleService', auxServiceId='spark_shuffle', configs={spark.authenticate=false, spark.shuffle.service.port=7337}, service=Spark}].

please help

Re: spark_shuffle fails nodemanager

Rayp — Thu, 15 Nov 2018 09:37:41 GMT

Issue was with spark authentication set up between spark and spar2 in cdh. default value of spark2 did not match to spark. once changed it resolved the issue spark.authenticate=true

question Re: spark_shuffle fails nodemanager in Archives of Support Questions (Read Only)

spark_shuffle fails nodemanager

Re: spark_shuffle fails nodemanager