HDFS is unable to change user/group in a cluster with Isilon

[#1]

SYMPTOMS:

When using HDFS commands to change ownership of a directory on a cluster backed with Isilon, it returns the following error:

[[email protected] ~]$ hadoop fs -chown -R hduser1:hdfs /user/hduser1 chown: changing ownership of '/user/hduser1': Failed to get id rec: 2:hduser1

Running "hdfs groups" command returns the following exception:

[[email protected] ~]$ hdfs groups hduser1 
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): bad protocol: org.apache.hadoop.tools.GetUserMappingsProtocol at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy8.getGroupsForUser(Unknown Source) at org.apache.hadoop.tools.protocolPB.GetUserMappingsProtocolClientSideTranslatorPB.getGroupsForUser(GetUserMappingsProtocolClientSideTranslatorPB.java:57)at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.hdfs.tools.GetGroups.main(GetGroups.java:96) 

The user is verified to be locally available on all nodes and in groups hadoop and hdfs.

[[email protected] ~]$ id -gn hduser1 
hadoop 
[[email protected] ~]$ id -Gn hduser1 
hadoop hdfs

ROOT CAUSE: User is not in Isilon

RESOLUTION: User must be added to Isilon.

isiloncluster1-1# isi auth groups create hduser1 --zone zone1 \ --provider local isiloncluster1-1 # isi auth users create hduser1 --primary-group hduser1 \ --zone zone1 --provider local \ --home-directory /ifs/isiloncluster1/zone1/hadoop/user/hduser1

Please reach out to EMC Isilon support if you have issues adding this user to Isilon.