permission denied user hive when creating external table in spark

[#1]

Hi ,

I am getting permission denied error when creating external table in hive context.

FYI : logged in as hdfs user

import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.orc._
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
val df = hiveContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load("/tmp/cars.csv")
val selectedData = df.select("year", "model")
selectedData.write.format("orc").option("header", "true").save("/tmp/newcars_orc_cust17") hiveContext.sql
("create external table newcars_orc_ext_cust17(year string,model string) stored as orc location '/tmp/newcars_orc_cust17'")
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.security.AccessControlException: Permission denied: user=hive, access=WRITE, inode="/tmp/newcars_orc_cust17":hdfs:hdfs:drwxr-xr-x 

[#2]

It looks like you are running Hive Jobs with the hive user, meaning your doAs-config (hive.server2.enable.doAs) is set to false. This flag ensures that jobs are always executed with the hive user instead of the user that is logged in.

You can find some information here

Is your cluster kerberized and do you have Ranger deployed?

If you change the owner to the folder to hive:hdfs it should work.

[#3]

@Jonas Straub I've the same issue. hive.server2.enable.doAs is set true in hive-site.xml. But still getting the same error.

[#4]

@Divya Gehlot try to run logged as hive user instead of hdfs user.

[#5]

@Divya Gehlot

Permission denied: user=hive, access=WRITE, inode="/tmp/newcars_orc_cust17

login as hdfs user

su - hdfs

hdfs dfs -s chown -R hive:hadoop /tmp/newcars_orc_cust17

[#6]

@Neeraj Sabharwal

In this above resolution which you have mentioned .. every time I have to change the permission .. Is there any settings which I can change ,

and give permission to the hive user for newly created hdfs files.

for instance:

selectedData.write.format("orc").option("header","true").save("/tmp/newcars_orc_cust17")

Thanks

[#7]

@Divya Gehlot Run it as Hive user.

[#8]

@Neeraj Sabharwal I've the same issue. I am running the spark application from 'mike' user account. The file exists in /user/mike/test folder and 'mike' user has read and write access to it. The error message is org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.security.AccessControlException: Permission denied: user=hive, access=WRITE, inode="/user/mike/test":mike:users:drwxr-xr-x

Looks like it is trying to access the file as 'hive' user although the mike user is running the spark application. I granted the write access to all the users to '/user/mike/test' and executed the application. It successfully created the external table.

I thought that the owner of the external table would be user 'hive', but it was 'mike'. The owner of the created table was set as the user executing the spark application. But the hdfs was accessed as 'hive' user.

What is actually happening?

[#9]

@Divya Gehlot has this been resolved? Please provide your solution or accept best answer.

[#10]

@Divya Gehlot Can you accept the best answer to close the thread?