Reply
Highlighted
New Contributor
Posts: 4
Registered: ‎01-13-2017

Re: Multiple Spark version on the same cluster

Yup, people should already be very carefull about it.

 

On the other hand, there are people with older CDH version with no Spark2 support available, or just trying to figure out if a vanilla(newer) version of spark has some bug(s) fixed, or whatever any other reason that works for them.

 

Regards.

New Contributor
Posts: 5
Registered: ‎02-08-2017

Re: Multiple Spark version on the same cluster

[ Edited ]

However this is a good explanation on how to run multiple spark installations on the same CDH, just adapting to other versions, so it's very valuable. 

 

One point though, does anything change kerberos-wise? I have done the same on different clusters, installing 1.6.3 into one non-kerberized CDH5.4 (Spark 1.3) and a kerberized CDH 5.5.3 (Spark 1.5.0).

Doing the same steps as in the non-kerberos installation (and issuing a ticket that allows me to spark-submit application with regular installed CDH Spark version), it fails like this: 

 

Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN xxxx for yyyy)

 

Could it be completed including steps necessary in a Kerberized installation? Thanks

Explorer
Posts: 6
Registered: ‎04-11-2017

Re: Multiple Spark version on the same cluster

How do I query hive tables from spark 2.0 . Could you share the steps.

Announcements