question Re: In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included in Archives of Support Questions (Read Only)

In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included

bandarusridhar1 — Fri, 24 Mar 2017 22:32:08 GMT

Hi,

With the Spark-SQL shell, kicking it off with settings along the lines of:

/usr/hdp/current/spark-client/bin/spark-sql --master yarn-client --queue abc --num-executors 5 --executor-memory 10g --executor-cores 10

Running the following command to create the table with minimal configuration:

create table [DB].[table_name] stored as orc as select….

Running into errors when attempting to grant select access to others for the table created in SQL Standard Authorization. Looking into it, notice:

Spark created table: owner: username@realm

Any regular Hive created table: owner: username

While using create table t1 as select from another table, the table t1 has owner name with the realm name included. Simple create table t1 without selecting from another table is working fine.

Re: In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included

deepesh1 — Sat, 25 Mar 2017 00:06:29 GMT

Is all the interaction from Spark shell or are you using HS2 beeline in the picture as well?

Re: In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included

bandarusridhar1 — Sat, 25 Mar 2017 01:01:50 GMT

@Deepesh

Everything is from Spark-SQL only.

Re: In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included

bandarusridhar1 — Sat, 25 Mar 2017 01:04:31 GMT

@Deepesh @Deepesh

Anyways this issue is confirmed as the bug, which is going to get fixed in Spark 2.1.+. Posted , so that someone can refer for future instead of spending their time.

Re: In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included

deepesh1 — Sat, 25 Mar 2017 03:29:25 GMT

Can you post the Spark JIRA so that others know which issue is tracking this?

Re: In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included

bandarusridhar1 — Wed, 29 Mar 2017 04:23:17 GMT

@Deepesh

Here is the https://issues.apache.org/jira/browse/SPARK-19970

Re: In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included

bandarusridhar1 — Sat, 03 Jun 2017 03:14:15 GMT

Unfortunately, spark is limited in terms of sqlstandard authorization. Spark isn't designed to work with hive sqlstandard authorizations due to which creating a table isn't creating default grants.

Here are two related JIRA's that mention the same.

https://issues.apache.org/jira/browse/SPARK-8321

https://issues.apache.org/jira/browse/SPARK-12008

You may create the necessary grants after table creation or Rely on storage based authorization i.e hdfs permissions for authorizations or use Ranger for hive authorizations .