Community Articles

Find and share helpful community-sourced technical articles.
Labels (1)
avatar
Expert Contributor

1. Goal

Security is one of fundamental features for enterprise adoption. Specifically, for SQL users, row/column-level access control is important. However, when a cluster is used as a data warehouse accessed by various user groups via different ways like Apache Spark™ 1.6/2.1 and Apache Hive, it is difficult to guarantee access control in a consistent way. In this article, we show the use of Apache Ranger™ to manage access control policy for Spark SQL. This enhancement is done by integrating Apache Ranger and Apache Spark via `spark-llap`. We also show how to provide finer grained access control(row/column level filtering and column masking) to Apache Spark.

2. Key Features

  • Shared Policies: The data in a cluster can be shared securely and consistently controlled by the shared access rules between Apache Spark and Apache Hive.
  • Audits: All security activities can be monitored and searched in a single place, i.e., Apache Ranger.
  • Resources management: Each user can use different queues while accessing the securely shared Hive data.

3. Environment

  • HDP 2.6.1.0 with Spark2, Hive and Ranger on a Kerberized cluster
  • SPARK-LLAP: 1.1.3-2.1

4. Assumption

4.1. HDFS Permission

For all use cases, make it sure that the permission of Hive warehouse is 700. It means normal users are unable to access the secured tables. In addition, make sure that `hive.warehouse.subdir.inherit.perms=true`. With this, newly created tables will inherit the permission, 700, by default.

$ hadoop fs -ls /apps/hive
Found 1 items
drwx------ - hive hdfs 0 2017-07-10 17:04 /apps/hive/warehouse

4.2. Interactive Query

Hive Interactive Query is needed to be enabled.

Screen Shot 2017-05-04 at 11.41.59 AM.png

4.3. Ranger Hive Plugin

Hive Ranger Plugin is needed to be enabled.

Screen Shot 2017-05-04 at 11.42.10 AM.png

4.4. User Principals

In this article, we will use two user principals, `billing` and `datascience`. `billing` principal can access all rows and columns while `datascience` principal can access some of filtered and masked data. You can use `hive` and `spark` principal instead.

5. Configurations

5.1. Find the existing configuration of your cluster

First of all, find the following five values from Hive configuration via Ambari for your cluster.

  • HiveServer2 Interactive Host name
    • You need a host name instead of JDBC URL using Zookeeper.
  • Spark Thrift Server Host
    • This value will be used during running example code.
  • The value for hive.llap.daemon.service.hosts

Screen Shot 2017-05-04 at 11.44.29 AM.png

  • The value for hive.zookeeper.quorum

Screen Shot 2017-05-04 at 11.45.07 AM.png

  • The value for hive.server2.authentication.kerberos.principal

Screen Shot 2017-05-04 at 11.45.51 AM.png

5.2. Setup HDFS

Set the following parameter.

  • `Custom core-site` → `hadoop.proxyuser.hive.hosts`
    • `*` will allow all hosts to submit Spark jobs with SPARK-LLAP.
*

5.3. Setup Hive

Set the following two parameters.

  • `Custom hive-interactive-site`
    • These are the same values with `hive.llap.daemon.keytab.file` and `hive.llap.daemon.service.principal` in the same `Custom hive-interactive-site`.
hive.llap.task.keytab.file=/etc/security/keytabs/hive.service.keytab
hive.llap.task.principal=hive/_HOST@EXAMPLE.COM

5.4. Setup Spark2

For Spark2 shells (spark-shell, pyspark, sparkR, spark-sql) and applications, setup the following configurations via Ambari and restart required services.

  • `Custom spark2-default`
spark.hadoop.hive.llap.daemon.service.hosts=the value of hive.llap.daemon.service.hosts
spark.hadoop.hive.zookeeper.quorum=The value of hive.zookeeper.quorum
spark.sql.hive.hiveserver2.jdbc.url=jdbc:hive2://YourHiveServer2HostName:10500/
spark.sql.hive.hiveserver2.jdbc.url.principal=The value of hive.server2.authentication.kerberos.principal

5.5. Setup Spark2 Thrift Server

For Spark2 Thrift Server, setup the following configurations via Ambari and restart required services.

  • `Advanced spark2-env` → `spark_thrift_cmd_opts`
--packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true

Note that this is one line.

  • `Custom spark2-thrift-sparkconf`
    • Note that `spark.sql.hive.hiveserver2.jdbc.url` additionally has `;hive.server2.proxy.user=${user}` for impersonation.
spark.hadoop.hive.llap.daemon.service.hosts=the value of hive.llap.daemon.service.hosts
spark.hadoop.hive.zookeeper.quorum=The value of hive.zookeeper.quorum
spark.sql.hive.hiveserver2.jdbc.url=jdbc:hive2://YourHiveServer2HostName:10500/;hive.server2.proxy.user=${user}
spark.sql.hive.hiveserver2.jdbc.url.principal=The value of hive.server2.authentication.kerberos.principal

6. Prepare database

Since normal users are unable to access Hive databases and tables until now due to HDFS permissions, use Beeline CLI with `hive` principal to prepare a database and a table for users who will use Spark-LLAP. Please note that the following is an example schema setup for demo scenario in this article. In case of audit-only scenarios, you don’t need to make new databases and tables. Instead, you can make all existing databases and tables accessible for all users.

$ kdestroy

$ kinit -kt /etc/security/keytabs/hive.service.keytab hive/`hostname -f`@EXAMPLE.COM

$ beeline -u "jdbc:hive2://YourHiveServerHost:10500/default;principal=hive/_HOST@EXAMPLE.COM" -e "CREATE DATABASE db_spark; USE db_spark; CREATE TABLE t_spark(name STRING, gender STRING); INSERT INTO t_spark VALUES ('Barack Obama', 'M'), ('Michelle Obama', 'F'), ('Hillary Clinton', 'F'), ('Donald Trump', 'M');"

7. Security Policies

To use a fine-grained access control, you need to setup Apache Ranger™ policies which rule Spark and Hive together seamlessly as a single control center.

7.1. Ranger Admin UI

Open `Ranger Admin UI`. The default login information is `admin/admin`. After login, `Access Manager` page shows nine service managers. The following screenshot means there exists HDFS / Hive / YARN policies. Since Spark shares the same policies with Hive, visit `Hive` among service managers.

Screen Shot 2017-05-04 at 11.48.21 AM.png

In Hive Service Manager page, there are three tabs corresponding three types of policies: `Access`, `Masking`, and `Row Level Filter`.

Screen Shot 2017-05-04 at 11.49.00 AM.png

Screen Shot 2017-05-04 at 11.49.03 AM.png

Screen Shot 2017-05-04 at 11.49.05 AM.png

7.2. Example Policies

Let’s make some policies for a user to access some rows and columns.

For examples,

  • Both `billing` and `datascience` principal can access `db_spark` database. The other databases are not allowed by default due to the HDFS permission.
  • `billing` principal can see all rows and columns of `t_spark` table in `db_spark` database.
  • `datascience` principal can see the first 4 characters of `name` field of `t_spark` table in `db_spark` database. The remaining part of `name` field will be masked.
  • `datascience` principal can see only the rows filtered by `gender=M` predicate.

7.2.1. Access policy in db_spark database

Name Table Column Select User Permissions
spark_access t_spark * billing Select
spark_access t_spark * datascience Select

Screen Shot 2017-05-04 at 11.58.53 AM.png

7.2.2. Masking policy in db_spark database

Name Table Column Select User Access Types Select Masking Option
spark_mask t_spark name datascience Select partial mask:'show first 4'

Screen Shot 2017-05-04 at 12.01.35 PM.png

7.2.3. Row Level Filter policy in db_spark database

Name Table Access Types Row Level Filter
spark_filter t_spark Select gender='M'

Screen Shot 2017-05-04 at 12.04.00 PM.png

7.2.4. HDFS policy for spark

In HDFS Ranger plugin, add a rule `spark_tmp` to allow all accesses on `/tmp`.

8. Target Scenarios

Case 1: Secure Spark Thrift Server

A user can access the Spark Thrift Server via beeline or Apache Zeppelin™. First, based on the kerberos principal, the user can see only the accessible data.

$ kdestroy

$ kinit billing/billing@EXAMPLE.COM

$ beeline -u "jdbc:hive2://YourSparkThriftServer:10016/db_spark;principal=hive/_HOST@EXAMPLE.COM" -e 'select * from db_spark.t_spark'

+------------------+---------+--+
| name | gender |
+------------------+---------+--+
| Barack Obama | M |
| Michelle Obama | F |
| Hillary Clinton | F |
| Donald Trump | M |
+------------------+---------+--+

$ kdestroy

$ kinit datascience/datascience@EXAMPLE.COM

$ beeline -u "jdbc:hive2://YourSparkThriftServer:10016/db_spark;principal=hive/_HOST@EXAMPLE.COM" -e 'select * from db_spark.t_spark'
+---------------+---------+--+
| name | gender |
+---------------+---------+--+
| Baraxx xxxxx | M |
| Donaxx xxxxx | M |
+---------------+---------+--+

Second, in case of Zeppelin, a proxy user name is used. You can see this youtube demo to see how Zeppelin works. The following example illustrates the usage of proxy user name with beeline. Zeppelin does the same thing via JDBC under the hood.

$ kdestroy

$ kinit -kt /etc/security/keytabs/hive.service.keytab hive/`hostname -f`@EXAMPLE.COM

$ beeline -u "jdbc:hive2://YourSparkThriftServerHost:10016/db_spark;principal=hive/_HOST@EXAMPLE.COM;hive.server2.proxy.user=billing" -e "select * from db_spark.t_spark"
+------------------+---------+--+
| name | gender |
+------------------+---------+--+
| Barack Obama | M |
| Michelle Obama | F |
| Hillary Clinton | F |
| Donald Trump | M |
+------------------+---------+--+

$ beeline -u "jdbc:hive2://YourSparkThriftServerHost:10016/db_spark;principal=hive/_HOST@EXAMPLE.COM;hive.server2.proxy.user=datascience" -e "select * from db_spark.t_spark"
+---------------+---------+--+
| name | gender |
+---------------+---------+--+
| Baraxx xxxxx | M |
| Donaxx xxxxx | M |
+---------------+---------+--+

Case 2: Shells

A user can run `spark-shell` or `pyspark` like the followings. Please note that the user can access own data sources in addition to the secure data source provided by `spark-llap`. For the next example, log in as the user `spark`.

  • spark-shell
$ kdestroy

$ kinit billing/billing@EXAMPLE.COM

$ SPARK_MAJOR_VERSION=2 spark-shell --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true
scala> sql("select * from db_spark.t_spark").show
+---------------+------+
| name|gender|
+---------------+------+
| Barack Obama| M|
| Michelle Obama| F|
|Hillary Clinton| F|
| Donald Trump| M|
+---------------+------+

$ kdestroy

$ kinit datascience/datascience@EXAMPLE.COM

$ SPARK_MAJOR_VERSION=2 spark-shell --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true
scala> sql("select * from db_spark.t_spark").show
+------------+------+
| name|gender|
+------------+------+
|Baraxx xxxxx| M|
|Donaxx xxxxx| M|
+------------+------+
  • pyspark
$ kdestroy

$ kinit billing/billing@EXAMPLE.COM

$ SPARK_MAJOR_VERSION=2 pyspark --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true

>>> sql("select * from db_spark.t_spark").show()
+---------------+------+
| name|gender|
+---------------+------+
| Barack Obama| M|
| Michelle Obama| F|
|Hillary Clinton| F|
| Donald Trump| M|
+---------------+------+

$ kdestroy

$ kinit datascience/datascience@EXAMPLE.COM

$ SPARK_MAJOR_VERSION=2 pyspark --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true

>>> sql("select * from db_spark.t_spark").show()
+------------+------+
| name|gender|
+------------+------+
|Baraxx xxxxx| M|
|Donaxx xxxxx| M|
+------------+------+
  • sparkR
$ kdestroy

$ kinit billing/billing@EXAMPLE.COM

$ SPARK_MAJOR_VERSION=2 sparkR --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true
> head(sql("select * from db_spark.t_spark"))
name gender
1 Barack Obama M
2 Michelle Obama F
3 Hillary Clinton F
4 Donald Trump M

$ kdestroy

$ kinit datascience/datascience@EXAMPLE.COM

$ SPARK_MAJOR_VERSION=2 sparkR --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true
> head(sql("select * from db_spark.t_spark"))
name gender
1 Baraxx xxxxx M
2 Donaxx xxxxx M
  • spark-sql
$ kdestroy

$ kinit billing/billing@EXAMPLE.COM

$ SPARK_MAJOR_VERSION=2 spark-sql --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true
spark-sql> select * from db_spark.t_spark;
Barack ObamaM
Michelle ObamaF
Hillary ClintonF
Donald TrumpM

$ kdestroy

$ kinit datascience/datascience@EXAMPLE.COM

$ SPARK_MAJOR_VERSION=2 spark-sql --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true
spark-sql> select * from db_spark.t_spark
Baraxx xxxxxM
Donaxx xxxxxM

Case 3: Applications

A user can submit his Spark job like the following. Like `spark-shell` scenario, the user can access own data sources in addition to the secure data source provided by `spark-llap`.

from pyspark.sql import SparkSession

spark = SparkSession \
 .builder \
 .appName("Spark LLAP SQL Python") \
 .enableHiveSupport() \
 .getOrCreate()

spark.sql("show databases").show()
spark.sql("select * from db_spark.t_spark").show()
spark.stop()

Launch the app with YARN client mode.

SPARK_MAJOR_VERSION=2 spark-submit --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.3-2.1 --repositories http://repo.hortonworks.com/content/groups/public --master yarn --deploy-mode client --conf spark.sql.hive.llap=true spark_llap_sql.py

YARN cluster mode will be supported in next HDP release. You can download the examples at spark_llap_sql.py, too.

Appendix

Support Matrix

Please refer https://github.com/hortonworks-spark/spark-llap/wiki/7.-Support-Matrix

Known Issues

Warning logs on CancellationException

If you see warning messages in Spark shells, you can turn off via `sc.setLogLevel` or `conf/log4j.properties`.

scala> sql("select * from db_spark.t_spark").show
...
17/03/09 22:06:26 WARN TaskSetManager: Stage 5 contains a task of very large size (248 KB). The maximum recommended task size is 100 KB.
17/03/09 22:06:27 WARN LlapProtocolClientProxy: RequestManager shutdown with error
java.util.concurrent.CancellationException
...
scala> sc.setLogLevel("ERROR")
scala> sql("select * from db_spark.t_spark").show
...
20,279 Views
Comments
avatar
New Contributor
hi,I have a problem,when set
  • `Advanced spark2-env` → `spark_thrift_cmd_opts`, it's value is your article write:
    1. --packages com.hortonworks.spark:spark-llap-assembly_2.11:1.1.1-2.1--repositories http://repo.hortonworks.com/content/groups/public --conf spark.sql.hive.llap=true
    2. when I set above value,start spark thriftserver throws errors. I need set addition settings in my cluster or my cluster's hosts.
    3. hope your answer,thanks!
avatar
Expert Contributor

Sorry for late response. Is your cluster connected to the internet to download the jar file? What error did you see?

avatar
Master Guru

The same problem here @Dongjoon Hyun, the cluster is not connected to the Internet, and browsing http://repo.hortonworks.com/content/groups/public/com/hortonworks/spark/spark-llap-assembly_2.11/1.1... returns no jars, only pom for Maven.

avatar
Expert Contributor

That's an assembly jar location for Spark `--packages`. If you want to a standalone jar for no-internet connection, please look at this.

http://repo.hortonworks.com/content/groups/public/com/hortonworks/spark/spark-llap_2.11/1.1.3-2.1/

avatar
New Contributor

Thank you for your reply,The problem is resolved,The problem's reason is download the jar file error.

In addition,I have a problem is Turn on Hive LLAP,Spark ThriftServer access Hive speed slow down。I try to modify some of the Hive llap related parameters.but no significant effect。I need your Professional advice and help。Can I get your email and get more help.Hope your quick answer,thanks!

avatar
Expert Contributor

For your emails, I already answered your email questions personally, and Hortonworks support team can help you for the further professional advice and help.

avatar
New Contributor

Thanks,I receive your reply.I will focus your blog article and github code about llap.

avatar
New Contributor

39422-6e548201-a008-4a1e-b7b0-c3e699cbba7b.png

spark thrift server jdbc error:

avatar
Expert Contributor

Hi, @chenhao chenhao

As you can see in the image, the error comes from Hive side, "Invalid function `get_splits`". You seems to use some old Hive. Please use the correct version of Hive in HDP.

avatar

Hello,

I followed the instruction with HDP 2.6.3.0, however, Spark2 Thrift Server stops right after starting it with the following error in /var/log/spark2/spark-hive-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-<HOSTNAME>.out:

Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with LLAP support because LLAP or Hive classes are not found.
        at org.apache.spark.sql.SparkSession$.isLLAPEnabled(SparkSession.scala:1104)
        at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$externalCatalogClassName(SharedState.scala:174)
        at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:95)
        at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:93)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:81)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Looks like this error indicates Spark2 Thrift Server fails to load org.apache.hadoop.hive.conf.HiveConf or org.apache.spark.sql.hive.llap.LlapSessionStateBuilder classes. I found com.hortonworks.spark_spark-llap_2.11-1.1.3-2.1.jar, which Spark2 Thrift Server is using, does not have org.apache.hadoop.hive.conf.HiveConf but shadehive.org.apache.hadoop.hive.conf.HiveConf

Can I ask if it's a bug? Can I also ask if there is a workaround?

Thank you in advance,

Mai Nakagawa