Yes all that is correct regarding how the CDH build works vs upstream and Kafka version. Although I admit I have not tried it directly it is my understanding that all this is so you can use security with Kafka and Spark Streaming. ... View more

Yes you will certainly need to provide access keys for S3 access to work. I don't think (?) that would be a solution to a VerifyError, which is a much lower-level error indicating corrupted builds. Yes, it's expected that AWS SDK dependencies were updated along with the new Spark version in CDH 5.7. I think the current version should depend on jets3t 0.9, which is the one you want. ... View more

That looks like a S3 library problem. The JVM says the bytecode itself is invalid. It is nothing to do with Spark per se. CDH does not support S3 although there's no particular reason it wouldn't work if you had the right libraries in place. ... View more

Typically you set this per job on the command line as args to spark-shell. If a setting is really something to establish as a default, you can update or point to a new, different spark-defaults.conf for your jobs. Advanced config snippets are for services, like the Spark history server, at least to my understanding. I'm not sure that would apply. ... View more

Heh, that is a large part of what dynamic allocation was meant for, so you could have a long running process that could only consume resources when it's active. and a shell sitting open is a prime example of that.   To some degree you can manage this via resource pools in YARN, and restrict a user, group or perhaps type of usage to a certain set of resources. This would be a pretty crude limit though, just a cap on the problem. Open shells would still keep resources.   Timing out shells is tricky because you lose work and state; that's probably pretty surprising. Really you want dynamic allocation for this. ... View more

Try "sc.emptyRDD[(String,Int)]"; currently the type of the RDD is not inferrable and so isn't obviously a match for the ones that it's unioned to later.   But, you really want to use SparkContext.union here to union many RDDs. Make a Seq of them and then call once to union them. ... View more