In Spark-SQL create table t1 as select from another table, the table t1 has owner name with the realm name included

[#1]

Hi,

With the Spark-SQL shell, kicking it off with settings along the lines of:

/usr/hdp/current/spark-client/bin/spark-sql --master yarn-client --queue abc --num-executors 5 --executor-memory 10g --executor-cores 10 

Running the following command to create the table with minimal configuration:

create table [DB].[table_name] stored as orc as select…. 

Running into errors when attempting to grant select access to others for the table created in SQL Standard Authorization. Looking into it, notice:

Spark created table: owner: username@realm

Any regular Hive created table: owner: username

While using create table t1 as select from another table, the table t1 has owner name with the realm name included. Simple create table t1 without selecting from another table is working fine.

[#2]

Is all the interaction from Spark shell or are you using HS2 beeline in the picture as well?

[#3]

@Deepesh

Everything is from Spark-SQL only.

[#4 · Accepted solution]

@Deepesh@Deepesh

Anyways this issue is confirmed as the bug, which is going to get fixed in Spark 2.1.+. Posted , so that someone can refer for future instead of spending their time.

[#5]

Can you post the Spark JIRA so that others know which issue is tracking this?

[#6]

@Deepesh

Here is the https://issues.apache.org/jira/browse/SPARK-19970

[#7]

Unfortunately, spark is limited in terms of sqlstandard authorization. Spark isn't designed to work with hive sqlstandard authorizations due to which creating a table isn't creating default grants.

Here are two related JIRA's that mention the same.

https://issues.apache.org/jira/browse/SPARK-8321

https://issues.apache.org/jira/browse/SPARK-12008

You may create the necessary grants after table creation or Rely on storage based authorization i.e hdfs permissions for authorizations or use Ranger for hive authorizations .