Reply
New Contributor
Posts: 2
Registered: ‎01-09-2015

Permissions issue writing to managed Hive partition using HCatLoader with different user than owner

Apache Pig version 0.12.0-cdh5.1.3

Hive 0.12.0-cdh5.1.3

I am trying to write a new partition to a managed Hive table using HCatStorer in Pig. Everything works fine when I am using the same user who created the table, but I get an error when trying to create a new partition using a different user. The HDFS permissions on the /user/hive/warehouse/table.db/ location appear to be fine. The pig script is able to create a new directory that would correspond to the partition, but seems to fail when trying to actually write data to that partition with the following error:

java.io.IOException: java.lang.reflect.InvocationTargetException
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:264)
    at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:253)
    at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:216)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:262)
    ... 5 more
Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding partition to metastore. Cause : org.apache.hadoop.security.AccessControlException: Permission denied
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:201)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:191)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5607)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5589)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:5545)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermissionInt(FSNamesystem.java:1559)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1540)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:542)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:401)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

    at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:704)
    at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:206)
    ... 10 more
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied

I am not sure what permissions could be affecting this. It appears to be an HCatalog issue, but all the documentation I could find indicates that permissions are controlled by HDFS permissions so there should be no issue creating the partition.

New Contributor
Posts: 3
Registered: ‎08-31-2017

Re: Permissions issue writing to managed Hive partition using HCatLoader with different user than ow

[ Edited ]

Hi,

I am facing same issue.

Were you able to resolve it?

 

thanks,

Rishit Shah

Cloudera Employee
Posts: 8
Registered: ‎04-03-2017

Re: Permissions issue writing to managed Hive partition using HCatLoader with different user than ow

Hi,

 

It looks like you are not able to use Hcatalog to access the Hive Metastore.

 

Can you please share what's the value you have set in hive proxy group?

 

## CM->Hive->Configuration->Search "hadoop.proxyuser.hive.groups"

 

I guess you might have added some users in it, Can you please confirm?

 

Regards

Nitish

Highlighted
New Contributor
Posts: 3
Registered: ‎08-31-2017

Re: Permissions issue writing to managed Hive partition using HCatLoader with different user than ow

[ Edited ]

Hi,

 

Thanks for reply.

The value in hadoop.proxyuser.hive.groups is blank in CM.

 

 

thanks,

Rishit Shah

Announcements