1. Goal
Security is one of fundamental features for enterprise adoption. Specifically, for SQL users, row/column-level access control is important. However, when a cluster is used as a data warehouse accessed by various user groups via different ways like Apache Spark™ 1.6/2.1 and Apache Hive, it is difficult to guarantee access control in a consistent way. In this article, we show the use of Apache Ranger™ to manage access control policy for Spark SQL. This enhancement is done by integrating Apache Ranger and Apache Spark via `spark-llap`. We also show how to provide finer grained access control(row/column level filtering and column masking) to Apache Spark.
2. Key Features
- Shared Policies: The data in a cluster can be shared securely and consistently controlled by the shared access rules between Apache Spark and Apache Hive.
- Audits: All security activities can be monitored and searched in a single place, i.e., Apache Ranger.
- Resources management: Each user can use different queues while accessing the securely shared Hive data.
3. Environment
- HDP 2.6.1.0 with Spark2, Hive and Ranger on a Kerberized cluster
- SPARK-LLAP: 1.1.3-2.1
4. Assumption
4.1. HDFS Permission
For all use cases, make it sure that the permission of Hive warehouse is 700. It means normal users are unable to access the secured tables. In addition, make sure that `hive.warehouse.subdir.inherit.perms=true`. With this, newly created tables will inherit the permission, 700, by default.
$ hadoop fs -ls /apps/hive Found 1 items drwx------ - hive hdfs 0 2017-07-10 17:04 /apps/hive/warehouse
4.2. Interactive Query
Hive Interactive Query is needed to be enabled.
4.3. Ranger Hive Plugin
Hive Ranger Plugin is needed to be enabled.
4.4. User Principals
In this article, we will use two user principals, `billing` and `datascience`. `billing` principal can access all rows and columns while `datascience` principal can access some of filtered and masked data. You can use `hive` and `spark` principal instead.
5. Configurations
5.1. Find the existing configuration of your cluster
First of all, find the following five values from Hive configuration via Ambari for your cluster.
- HiveServer2 Interactive Host name
- You need a host name instead of JDBC URL using Zookeeper.
- Spark Thrift Server Host
- This value will be used during running example code.
- The value for hive.llap.daemon.service.hosts
- The value for hive.zookeeper.quorum
- The value for hive.server2.authentication.kerberos.principal
5.2. Setup HDFS
Set the following parameter.
- `Custom core-site` → `hadoop.proxyuser.hive.hosts`
- `*` will allow all hosts to submit Spark jobs with SPARK-LLAP.
*
5.3. Setup Hive
Set the following two parameters.
- `Custom hive-interactive-site`
- These are the same values with `hive.llap.daemon.keytab.file` and `hive.llap.daemon.service.principal` in the same `Custom hive-interactive-site`.
hive.llap.task.keytab.file=/etc/security/keytabs/hive.service.keytab hive.llap.task.principal=hive/_HOST@EXAMPLE.COM
5.4. Setup Spark2
For Spark2 shells (spark-shell, pyspark, sparkR, spark-sql) and applications, setup the following configurations via Ambari and restart required services.
- `Custom spark2-default`
spark.hadoop.hive.llap.daemon.service.hosts=the value of hive.llap.daemon.service.hosts spark.hadoop.hive.zookeeper.quorum=The value of hive.zookeeper.quorum spark.sql.hive.hiveserver2.jdbc.url=jdbc:hive2://YourHiveServer2HostName:10500/ spark.sql.hive.hiveserver2.jdbc.url.principal=The value of hive.server2.authentication.kerberos.principal
5.5. Setup Spark2 Thrift Server
For Spark2 Thrift Server, setup the following configurations via Ambari and restart required services.
- `Advanced spark2-env` → `spark_thrift_cmd_opts`
--packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true
Note that this is one line.
- `Custom spark2-thrift-sparkconf`
- Note that `spark.sql.hive.hiveserver2.jdbc.url` additionally has `;hive.server2.proxy.user=${user}` for impersonation.
spark.hadoop.hive.llap.daemon.service.hosts=the value of hive.llap.daemon.service.hosts
spark.hadoop.hive.zookeeper.quorum=The value of hive.zookeeper.quorum
spark.sql.hive.hiveserver2.jdbc.url=jdbc:hive2://YourHiveServer2HostName:10500/;hive.server2.proxy.user=${user}
spark.sql.hive.hiveserver2.jdbc.url.principal=The value of hive.server2.authentication.kerberos.principal
6. Prepare database
Since normal users are unable to access Hive databases and tables until now due to HDFS permissions, use Beeline CLI with `hive` principal to prepare a database and a table for users who will use Spark-LLAP. Please note that the following is an example schema setup for demo scenario in this article. In case of audit-only scenarios, you don’t need to make new databases and tables. Instead, you can make all existing databases and tables accessible for all users.
$ kdestroy
$ kinit -kt /etc/security/keytabs/hive.service.keytab hive/`hostname -f`@EXAMPLE.COM
$ beeline -u "jdbc:hive2://YourHiveServerHost:10500/default;principal=hive/_HOST@EXAMPLE.COM" -e "CREATE DATABASE db_spark; USE db_spark; CREATE TABLE t_spark(name STRING, gender STRING); INSERT INTO t_spark VALUES ('Barack Obama', 'M'), ('Michelle Obama', 'F'), ('Hillary Clinton', 'F'), ('Donald Trump', 'M');"
7. Security Policies
To use a fine-grained access control, you need to setup Apache Ranger™ policies which rule Spark and Hive together seamlessly as a single control center.
7.1. Ranger Admin UI
Open `Ranger Admin UI`. The default login information is `admin/admin`. After login, `Access Manager` page shows nine service managers. The following screenshot means there exists HDFS / Hive / YARN policies. Since Spark shares the same policies with Hive, visit `Hive` among service managers.
In Hive Service Manager page, there are three tabs corresponding three types of policies: `Access`, `Masking`, and `Row Level Filter`.
7.2. Example Policies
Let’s make some policies for a user to access some rows and columns.
For examples,
- Both `billing` and `datascience` principal can access `db_spark` database. The other databases are not allowed by default due to the HDFS permission.
- `billing` principal can see all rows and columns of `t_spark` table in `db_spark` database.
- `datascience` principal can see the first 4 characters of `name` field of `t_spark` table in `db_spark` database. The remaining part of `name` field will be masked.
- `datascience` principal can see only the rows filtered by `gender=M` predicate.
7.2.1. Access policy in db_spark database
| Name | Table | Column | Select User | Permissions |
| spark_access | t_spark | * | billing | Select |
| spark_access | t_spark | * | datascience | Select |
7.2.2. Masking policy in db_spark database
| Name | Table | Column | Select User | Access Types | Select Masking Option |
| spark_mask | t_spark | name | datascience | Select | partial mask:'show first 4' |
7.2.3. Row Level Filter policy in db_spark database
| Name | Table | Access Types | Row Level Filter |
| spark_filter | t_spark | Select | gender='M' |
7.2.4. HDFS policy for spark
In HDFS Ranger plugin, add a rule `spark_tmp` to allow all accesses on `/tmp`.
8. Target Scenarios
Case 1: Secure Spark Thrift Server
A user can access the Spark Thrift Server via beeline or Apache Zeppelin™. First, based on the kerberos principal, the user can see only the accessible data.
$ kdestroy $ kinit billing/billing@EXAMPLE.COM $ beeline -u "jdbc:hive2://YourSparkThriftServer:10016/db_spark;principal=hive/_HOST@EXAMPLE.COM" -e 'select * from db_spark.t_spark' +------------------+---------+--+ | name | gender | +------------------+---------+--+ | Barack Obama | M | | Michelle Obama | F | | Hillary Clinton | F | | Donald Trump | M | +------------------+---------+--+ $ kdestroy $ kinit datascience/datascience@EXAMPLE.COM $ beeline -u "jdbc:hive2://YourSparkThriftServer:10016/db_spark;principal=hive/_HOST@EXAMPLE.COM" -e 'select * from db_spark.t_spark' +---------------+---------+--+ | name | gender | +---------------+---------+--+ | Baraxx xxxxx | M | | Donaxx xxxxx | M | +---------------+---------+--+
Second, in case of Zeppelin, a proxy user name is used. You can see this youtube demo to see how Zeppelin works. The following example illustrates the usage of proxy user name with beeline. Zeppelin does the same thing via JDBC under the hood.
$ kdestroy $ kinit -kt /etc/security/keytabs/hive.service.keytab hive/`hostname -f`@EXAMPLE.COM $ beeline -u "jdbc:hive2://YourSparkThriftServerHost:10016/db_spark;principal=hive/_HOST@EXAMPLE.COM;hive.server2.proxy.user=billing" -e "select * from db_spark.t_spark" +------------------+---------+--+ | name | gender | +------------------+---------+--+ | Barack Obama | M | | Michelle Obama | F | | Hillary Clinton | F | | Donald Trump | M | +------------------+---------+--+ $ beeline -u "jdbc:hive2://YourSparkThriftServerHost:10016/db_spark;principal=hive/_HOST@EXAMPLE.COM;hive.server2.proxy.user=datascience" -e "select * from db_spark.t_spark" +---------------+---------+--+ | name | gender | +---------------+---------+--+ | Baraxx xxxxx | M | | Donaxx xxxxx | M | +---------------+---------+--+
Case 2: Shells
A user can run `spark-shell` or `pyspark` like the followings. Please note that the user can access own data sources in addition to the secure data source provided by `spark-llap`. For the next example, log in as the user `spark`.
- spark-shell
$ kdestroy $ kinit billing/billing@EXAMPLE.COM $ SPARK_MAJOR_VERSION=2 spark-shell --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true scala> sql("select * from db_spark.t_spark").show +---------------+------+ | name|gender| +---------------+------+ | Barack Obama| M| | Michelle Obama| F| |Hillary Clinton| F| | Donald Trump| M| +---------------+------+ $ kdestroy $ kinit datascience/datascience@EXAMPLE.COM $ SPARK_MAJOR_VERSION=2 spark-shell --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true scala> sql("select * from db_spark.t_spark").show +------------+------+ | name|gender| +------------+------+ |Baraxx xxxxx| M| |Donaxx xxxxx| M| +------------+------+
- pyspark
$ kdestroy $ kinit billing/billing@EXAMPLE.COM $ SPARK_MAJOR_VERSION=2 pyspark --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true >>> sql("select * from db_spark.t_spark").show() +---------------+------+ | name|gender| +---------------+------+ | Barack Obama| M| | Michelle Obama| F| |Hillary Clinton| F| | Donald Trump| M| +---------------+------+ $ kdestroy $ kinit datascience/datascience@EXAMPLE.COM $ SPARK_MAJOR_VERSION=2 pyspark --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true >>> sql("select * from db_spark.t_spark").show() +------------+------+ | name|gender| +------------+------+ |Baraxx xxxxx| M| |Donaxx xxxxx| M| +------------+------+
- sparkR
$ kdestroy $ kinit billing/billing@EXAMPLE.COM $ SPARK_MAJOR_VERSION=2 sparkR --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true > head(sql("select * from db_spark.t_spark")) name gender 1 Barack Obama M 2 Michelle Obama F 3 Hillary Clinton F 4 Donald Trump M $ kdestroy $ kinit datascience/datascience@EXAMPLE.COM $ SPARK_MAJOR_VERSION=2 sparkR --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true > head(sql("select * from db_spark.t_spark")) name gender 1 Baraxx xxxxx M 2 Donaxx xxxxx M
- spark-sql
$ kdestroy $ kinit billing/billing@EXAMPLE.COM $ SPARK_MAJOR_VERSION=2 spark-sql --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true spark-sql> select * from db_spark.t_spark; Barack ObamaM Michelle ObamaF Hillary ClintonF Donald TrumpM $ kdestroy $ kinit datascience/datascience@EXAMPLE.COM $ SPARK_MAJOR_VERSION=2 spark-sql --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true spark-sql> select * from db_spark.t_spark Baraxx xxxxxM Donaxx xxxxxM
Case 3: Applications
A user can submit his Spark job like the following. Like `spark-shell` scenario, the user can access own data sources in addition to the secure data source provided by `spark-llap`.
from pyspark.sql import SparkSession
spark = SparkSession \
.builder \
.appName("Spark LLAP SQL Python") \
.enableHiveSupport() \
.getOrCreate()
spark.sql("show databases").show()
spark.sql("select * from db_spark.t_spark").show()
spark.stop()
Launch the app with YARN client mode.
SPARK_MAJOR_VERSION=2 spark-submit --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --master yarn --deploy-mode client --conf spark.sql.hive.llap=true spark_llap_sql.py
YARN cluster mode will be supported in next HDP release. You can download the examples at spark_llap_sql.py, too.
Appendix
Support Matrix
Please refer https://github.com/hortonworks-spark/spark-llap/wiki/7.-Support-Matrix
Known Issues
Warning logs on CancellationException
If you see warning messages in Spark shells, you can turn off via `sc.setLogLevel` or `conf/log4j.properties`.
scala> sql("select * from db_spark.t_spark").show
...
17/03/09 22:06:26 WARN TaskSetManager: Stage 5 contains a task of very large size (248 KB). The maximum recommended task size is 100 KB.
17/03/09 22:06:27 WARN LlapProtocolClientProxy: RequestManager shutdown with error
java.util.concurrent.CancellationException
...
scala> sc.setLogLevel("ERROR")
scala> sql("select * from db_spark.t_spark").show
...
