Reply
Explorer
Posts: 10
Registered: ‎02-12-2014

Spark and Sentry

Hi Folks,

 

Does Spark or SparkSQL supports Sentry?

Posts: 1,566
Kudos: 287
Solutions: 240
Registered: ‎07-31-2013

Re: Spark and Sentry

SparkSQL accesses its metadata via the HMS directly, and does not go through a HS2, so it does not truly get covered fully by Sentry. However, in a Sentry setup the HMS is write-protected via the Sentry Authz Plugin added on it, so DDLs are still protected against, but users can still view all metadata (i.e. they can run SHOW TABLES, SHOW DATABASES, etc. and retrieve full listing [1]).

With Sentry HMS plugin and Sentry HDFS ACL Sync enabled, access to tables' data by Spark programs would be limited to the same rules as your Beeline/other Hive clients would.

[1] - https://github.com/cloudera/sentry/blob/cdh5.7.0-release/sentry-binding/sentry-binding-hive/src/main...
Backline Customer Operations Engineer
Explorer
Posts: 31
Registered: ‎05-09-2017

Re: Spark and Sentry

[ Edited ]

@Harsh J

 

we recently installed spark2 in our CDH 5.13.0 cluster. Our tests show that sentry roles are not being applied . 

 

so are you saying that we need Sentry HMS plugin and Sentry HDFS ACL Sync enabled ? for spark programs to have Sentry roles enabled ?

 

Highlighted
Posts: 1,566
Kudos: 287
Solutions: 240
Registered: ‎07-31-2013

Re: Spark and Sentry

If Sentry is enabled, the HMS plugin should already be applied, so metadata write/modification queries are already authorized no matter where they come from (Hive, Impala, Spark, etc.).

 

The direct HDFS access to table files that Spark requires can only be granted to the end-users if you have Sentry HDFS ACL Sync enabled, such that the ACLs are applied for all granted role groups on the HDFS level automatically, allowing normal read/write access.

Backline Customer Operations Engineer
Explorer
Posts: 31
Registered: ‎05-09-2017

Re: Spark and Sentry

Thanks @Harsh J for your response. 

 

Does Sentry translate server level privileges to HDFS ACL's or does it just translate table privileges ? 

 

 

 

Announcements