Created 11-20-2017 04:45 PM
I am unable to load data into hive from a file that exists on hdfs using hive interactively. If I place the user in the hdfs group, this works, but, of course, grants way more privileges than I want on hdfs for a basic user.
Can interactive hive commands access HDFS without being a member of the group?
========================================================================
user@server:~$ hdfs dfs -put /home/user/test99.csv /user/user/test99.csv
user@server:~$ hive log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender. Logging initialized using configuration in file:/etc/hive/2.6.0.3-8/0/hive-log4j.properties
hive> LOAD DATA INPATH '/user/user/test99.csv' INTO TABLE test96;
Loading data to table default.test96 Failed with exception org.apache.hadoop.security.AccessControlException: User null does not belong to hdfs
at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:89)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1846)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:827)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:475)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask hive>
Created 11-26-2017 09:20 AM
Your user does not have permissions to write to HDFS.
Try to run the query with the user that belongs to hadoop/hdfs group.
Created 11-29-2017 03:19 PM
My user has access to HDFS, but only his own space. As I mentioned in the question, I know it works when I put the regular user into the HDFS group. I do not want to do this, because it grants the user access to everyone's HDFS space, just like the hdfs user. I want to get this working without the user being a member of the hdfs group. Is this possible?