Support Questions

Find answers, ask questions, and share your expertise

Unable to load data into hive from file on hdfs - User null does not belong to hdfs

avatar
Explorer

I am unable to load data into hive from a file that exists on hdfs using hive interactively. If I place the user in the hdfs group, this works, but, of course, grants way more privileges than I want on hdfs for a basic user.

Can interactive hive commands access HDFS without being a member of the group?

========================================================================

user@server:~$ hdfs dfs -put /home/user/test99.csv /user/user/test99.csv


user@server:~$ hive
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/2.6.0.3-8/0/hive-log4j.properties


hive> LOAD DATA INPATH '/user/user/test99.csv' INTO TABLE test96;


Loading data to table default.test96
Failed with exception org.apache.hadoop.security.AccessControlException: User null does not belong to hdfs


at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:89)


at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1846)


at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:827)


at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:475)


at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)


at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)


at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)


at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)


at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345)



FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
hive>

2 REPLIES 2

avatar
Explorer

Your user does not have permissions to write to HDFS.

Try to run the query with the user that belongs to hadoop/hdfs group.

avatar
Explorer

My user has access to HDFS, but only his own space. As I mentioned in the question, I know it works when I put the regular user into the HDFS group. I do not want to do this, because it grants the user access to everyone's HDFS space, just like the hdfs user. I want to get this working without the user being a member of the hdfs group. Is this possible?