02-16-2017 06:29 AM
Yup, people should already be very carefull about it.
On the other hand, there are people with older CDH version with no Spark2 support available, or just trying to figure out if a vanilla(newer) version of spark has some bug(s) fixed, or whatever any other reason that works for them.
04-18-2017 04:09 AM - edited 04-18-2017 04:11 AM
However this is a good explanation on how to run multiple spark installations on the same CDH, just adapting to other versions, so it's very valuable.
One point though, does anything change kerberos-wise? I have done the same on different clusters, installing 1.6.3 into one non-kerberized CDH5.4 (Spark 1.3) and a kerberized CDH 5.5.3 (Spark 1.5.0).
Doing the same steps as in the non-kerberos installation (and issuing a ticket that allows me to spark-submit application with regular installed CDH Spark version), it fails like this:
Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN xxxx for yyyy)
Could it be completed including steps necessary in a Kerberized installation? Thanks