Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Multiple Spark version on the same cluster

Solved Go to solution
Highlighted

Re: Multiple Spark version on the same cluster

New Contributor

Yup, people should already be very carefull about it.

 

On the other hand, there are people with older CDH version with no Spark2 support available, or just trying to figure out if a vanilla(newer) version of spark has some bug(s) fixed, or whatever any other reason that works for them.

 

Regards.

Re: Multiple Spark version on the same cluster

New Contributor

However this is a good explanation on how to run multiple spark installations on the same CDH, just adapting to other versions, so it's very valuable. 

 

One point though, does anything change kerberos-wise? I have done the same on different clusters, installing 1.6.3 into one non-kerberized CDH5.4 (Spark 1.3) and a kerberized CDH 5.5.3 (Spark 1.5.0).

Doing the same steps as in the non-kerberos installation (and issuing a ticket that allows me to spark-submit application with regular installed CDH Spark version), it fails like this: 

 

Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN xxxx for yyyy)

 

Could it be completed including steps necessary in a Kerberized installation? Thanks

Re: Multiple Spark version on the same cluster

How do I query hive tables from spark 2.0 . Could you share the steps.

Don't have an account?
Coming from Hortonworks? Activate your account here