Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark 2.4 on CDH 5.13.1

spark 2.4 on CDH 5.13.1

Contributor

Hi All,

we have spark 2.2 running in the cluster. Is there any way to install spark 2.4 by not touching the spark 2.2. downloaded the spark-2.4.4-bin-hadoop2.6.tgz onto one machine and I can able to launch the spark2-shell in local mode with 2.4.4 successfully. But i cannot succeed in the yarn mode. Can someone suggest the spark-defaults and spark-env.sh conf settings. 

Getting the below error: 

spark2-shell --master yarn
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/12/13 11:55:02 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/12/13 11:55:05 ERROR spark.SparkContext: Error initializing SparkContext.
java.io.IOException: java.net.UnknownHostException: namenode01;namenode02
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:815)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)

 

Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here