Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark and Sentry

Highlighted

Spark and Sentry

Hi Folks,

 

Does Spark or SparkSQL supports Sentry?

5 REPLIES 5

Re: Spark and Sentry

Master Guru
SparkSQL accesses its metadata via the HMS directly, and does not go through a HS2, so it does not truly get covered fully by Sentry. However, in a Sentry setup the HMS is write-protected via the Sentry Authz Plugin added on it, so DDLs are still protected against, but users can still view all metadata (i.e. they can run SHOW TABLES, SHOW DATABASES, etc. and retrieve full listing [1]).

With Sentry HMS plugin and Sentry HDFS ACL Sync enabled, access to tables' data by Spark programs would be limited to the same rules as your Beeline/other Hive clients would.

[1] - https://github.com/cloudera/sentry/blob/cdh5.7.0-release/sentry-binding/sentry-binding-hive/src/main...

Re: Spark and Sentry

Expert Contributor

@Harsh J

 

we recently installed spark2 in our CDH 5.13.0 cluster. Our tests show that sentry roles are not being applied . 

 

so are you saying that we need Sentry HMS plugin and Sentry HDFS ACL Sync enabled ? for spark programs to have Sentry roles enabled ?

 

Re: Spark and Sentry

Master Guru

If Sentry is enabled, the HMS plugin should already be applied, so metadata write/modification queries are already authorized no matter where they come from (Hive, Impala, Spark, etc.).

 

The direct HDFS access to table files that Spark requires can only be granted to the end-users if you have Sentry HDFS ACL Sync enabled, such that the ACLs are applied for all granted role groups on the HDFS level automatically, allowing normal read/write access.

Re: Spark and Sentry

Expert Contributor

Thanks @Harsh J for your response. 

 

Does Sentry translate server level privileges to HDFS ACL's or does it just translate table privileges ? 

 

 

 

Re: Spark and Sentry

New Contributor

@Harsh J Create table from sparkSql is failing with authorization denied for create table.
Where as the same command succeeded from Beeline.

How can we create table through spark?