Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark 2.4 on CDH 5.13.1

spark 2.4 on CDH 5.13.1


Hi All,

we have spark 2.2 running in the cluster. Is there any way to install spark 2.4 by not touching the spark 2.2. downloaded the spark-2.4.4-bin-hadoop2.6.tgz onto one machine and I can able to launch the spark2-shell in local mode with 2.4.4 successfully. But i cannot succeed in the yarn mode. Can someone suggest the spark-defaults and conf settings. 

Getting the below error: 

spark2-shell --master yarn
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/12/13 11:55:02 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/12/13 11:55:05 ERROR spark.SparkContext: Error initializing SparkContext. namenode01;namenode02
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(



Don't have an account?
Coming from Hortonworks? Activate your account here