Created 12-18-2015 06:09 AM
I am getting permission denied error when creating external table in hive context.
FYI : logged in as hdfs user
import org.apache.spark.sql.hive.HiveContext import org.apache.spark.sql.hive.orc._ val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) val df = hiveContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load("/tmp/cars.csv") val selectedData = df.select("year", "model") selectedData.write.format("orc").option("header", "true").save("/tmp/newcars_orc_cust17") hiveContext.sql ("create external table newcars_orc_ext_cust17(year string,model string) stored as orc location '/tmp/newcars_orc_cust17'") org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.security.AccessControlException: Permission denied: user=hive, access=WRITE, inode="/tmp/newcars_orc_cust17":hdfs:hdfs:drwxr-xr-x
Created 12-18-2015 07:19 AM
It looks like you are running Hive Jobs with the hive user, meaning your doAs-config (hive.server2.enable.doAs) is set to false. This flag ensures that jobs are always executed with the hive user instead of the user that is logged in.
You can find some information here
Is your cluster kerberized and do you have Ranger deployed?
If you change the owner to the folder to hive:hdfs it should work.
Created 03-30-2016 10:02 AM
@Jonas Straub I've the same issue. hive.server2.enable.doAs is set true in hive-site.xml. But still getting the same error.
Created 12-18-2015 10:31 AM
Created 12-18-2015 01:13 PM
Permission denied: user=hive, access=WRITE, inode="/tmp/newcars_orc_cust17
login as hdfs user
su - hdfs
hdfs dfs -s chown -R hive:hadoop /tmp/newcars_orc_cust17
Created 12-21-2015 02:22 AM
In this above resolution which you have mentioned .. every time I have to change the permission .. Is there any settings which I can change ,
and give permission to the hive user for newly created hdfs files.
Created 12-21-2015 03:48 AM
Created 04-01-2016 07:54 AM
@Neeraj Sabharwal I've the same issue. I am running the spark application from 'mike' user account. The file exists in /user/mike/test folder and 'mike' user has read and write access to it. The error message is org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.security.AccessControlException: Permission denied: user=hive, access=WRITE, inode="/user/mike/test":mike:users:drwxr-xr-x
Looks like it is trying to access the file as 'hive' user although the mike user is running the spark application. I granted the write access to all the users to '/user/mike/test' and executed the application. It successfully created the external table.
I thought that the owner of the external table would be user 'hive', but it was 'mike'. The owner of the created table was set as the user executing the spark application. But the hdfs was accessed as 'hive' user.
What is actually happening?
Created 02-03-2016 02:26 AM
Created 02-16-2016 01:28 AM