Spark and Sentry

[#1]

Hi Folks,

Does Spark or SparkSQL supports Sentry?

[#2]

SparkSQL accesses its metadata via the HMS directly, and does not go through a HS2, so it does not truly get covered fully by Sentry. However, in a Sentry setup the HMS is write-protected via the Sentry Authz Plugin added on it, so DDLs are still protected against, but users can still view all metadata (i.e. they can run SHOW TABLES, SHOW DATABASES, etc. and retrieve full listing [1]).

With Sentry HMS plugin and Sentry HDFS ACL Sync enabled, access to tables' data by Spark programs would be limited to the same rules as your Beeline/other Hive clients would.

[1] - https://github.com/cloudera/sentry/blob/cdh5.7.0-release/sentry-binding/sentry-binding-hive/src/main...

[#3]

@Harsh J

we recently installed spark2 in our CDH 5.13.0 cluster. Our tests show that sentry roles are not being applied . 

so are you saying that we need Sentry HMS plugin and Sentry HDFS ACL Sync enabled ? for spark programs to have Sentry roles enabled ?

[#4]

If Sentry is enabled, the HMS plugin should already be applied, so metadata write/modification queries are already authorized no matter where they come from (Hive, Impala, Spark, etc.).

The direct HDFS access to table files that Spark requires can only be granted to the end-users if you have Sentry HDFS ACL Sync enabled, such that the ACLs are applied for all granted role groups on the HDFS level automatically, allowing normal read/write access.

[#5]

Thanks @Harsh J for your response. 

Does Sentry translate server level privileges to HDFS ACL's or does it just translate table privileges ? 

[#6]

@Harsh J Create table from sparkSql is failing with authorization denied for create table.
Where as the same command succeeded from Beeline.

How can we create table through spark?