We don't support Spark on Hive, try Hive on Tez. We do plan to support SparkSQL when Spark 1.5.1 becomes GA on HDP. ... View more

There are multiple Spark to HBase connector. See an example on HDP http://stackoverflow.com/questions/25040709/how-to-read-from-hbase-using-spark But the existing connectors don't leverage HBase for Scans. We are building a more efficient Spark Hbase connector. ... View more

Spark reads from HDFS and sends jobs to YARN. So security for both HDFS, YARN managed by Ranger works with Spark. From security point of view this is very similar to MR jobs being run on YARN. Since Spark reads from HDFS using HDFS client, the HDFS TDE feature is transparent to Spark and with right key permissions for user running Spark job, there is nothing in Spark to configure. Knox isn’t yet relevant to Spark. In future when we have a REST API for Spark, we will integrate Knox with it. ... View more

Added a couple of sentences of clarification to get folks to Spark 1.4.1. If you have an HDP 2.3.0 cluster, it came with Spark 1.3.1, you can either upgrade the entire cluster withAmbari to 2.3.2 to get Spark 1.4.1 or only manually upgrade Spark to 1.4.1 ... View more

SparkSQL will be out of tech preview when Spark 1.5.1 is GA on HDP. Until then it is in TP. ... View more