Support Questions
Find answers, ask questions, and share your expertise

spark 2.4 on CDH 5.13.1

spark 2.4 on CDH 5.13.1

Contributor

Hi All,

we have spark 2.2 running in the cluster. Is there any way to install spark 2.4 by not touching the spark 2.2. downloaded the spark-2.4.4-bin-hadoop2.6.tgz onto one machine and I can able to launch the spark2-shell in local mode with 2.4.4 successfully. But i cannot succeed in the yarn mode. Can someone suggest the spark-defaults and spark-env.sh conf settings. 

Getting the below error: 

spark2-shell --master yarn
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/12/13 11:55:02 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/12/13 11:55:05 ERROR spark.SparkContext: Error initializing SparkContext.
java.io.IOException: java.net.UnknownHostException: namenode01;namenode02
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:815)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)

 

Thanks