question Re: What are the best practices to secure spark on HDP 2.3 in Archives of Support Questions (Read Only)

What are the best practices to secure spark on HDP 2.3

Eran — Sat, 31 Oct 2015 08:08:53 GMT

Re: What are the best practices to secure spark on HDP 2.3

nsabharwal — Sat, 31 Oct 2015 08:18:39 GMT

Re: What are the best practices to secure spark on HDP 2.3

Eran — Wed, 04 Nov 2015 03:59:13 GMT

So the real problem in our documentation - Best practices, has a conflicting message "use HiveContext (instead of SQLContext) whenever possible" but In YARN cluster mode with Kerberos, use SQLContext !

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_spark-guide/content/ch_practices-spark.html

Re: What are the best practices to secure spark on HDP 2.3

nsabharwal — Wed, 04 Nov 2015 04:05:34 GMT

@eorgad@hortonworks.com because "In YARN cluster mode with Kerberos, use SQLContext. HiveContext is not supported on YARN in a Kerberos-enabled cluster."

Re: What are the best practices to secure spark on HDP 2.3

ldaluz — Wed, 04 Nov 2015 04:14:16 GMT

@Neeraj The docs are conflicting. It is recommending "use HiveContext" in secure mode (with kerberos): http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_spark-guide/content/ch_installing-kerb-spark.html#spark-kerb-access-hive and under best practices it states that it is not supported with Kerberos: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_spark-guide/content/ch_practices-spark.html

Re: What are the best practices to secure spark on HDP 2.3

vshukla — Wed, 04 Nov 2015 04:56:30 GMT

This is a doc bug, filed BUG-47289

Use HiveContext and it should work in Kerberized Cluster with HDP 2.3.2

Re: What are the best practices to secure spark on HDP 2.3

vshukla — Wed, 04 Nov 2015 04:57:24 GMT

Spark reads from HDFS and sends jobs to YARN. So security for both HDFS, YARN managed by Ranger works with Spark. From security point of view this is very similar to MR jobs being run on YARN.

Since Spark reads from HDFS using HDFS client, the HDFS TDE feature is transparent to Spark and with right key permissions for user running Spark job, there is nothing in Spark to configure.

Knox isn’t yet relevant to Spark. In future when we have a REST API for Spark, we will integrate Knox with it.