App Timeline Server and History Server start failed with the same reason.

[#1]

Hi,

I am working on deploy HDP 2.6.4.0 with Isilon 8.1.0.1. Now App Timeline Server and History Server start failed with similar errors:

History Server:

File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 436, in _set_owner self.util.run_command(self.main_resource.resource.target, 'SETOWNER', method='PUT', owner=owner, group=group, assertable_result=False) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command raise Fail(err_msg)

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/tmp/entity-file-history/active?op=SETOWNER&user.name=hdfs&...'' returned status_code=403.

{ "RemoteException": { "exception": "AccessControlException", "javaClassName": "org.apache.hadoop.security.AccessControlException", "message": "Permission denied: user=hdfs, access=WRITE, path=\"/tmp/entity-file-history/active\"" } }

App Timeline Server:

File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 436, in _set_owner self.util.run_command(self.main_resource.resource.target, 'SETOWNER', method='PUT', owner=owner, group=group, assertable_result=False) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command raise Fail(err_msg)

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoo...'' returned status_code=403.

{ "RemoteException": { "exception": "AccessControlException", "javaClassName": "org.apache.hadoop.security.AccessControlException", "message": "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\"" } }

I am not sure which user is used here to run "SETOWNER" action defined in "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py".

Is there anybody meet such this issue?

Thank you!

[#2]

@Alex Yin

The error says the username = hdfs

{ "RemoteException": { "exception": "AccessControlException", 
"javaClassName": "org.apache.hadoop.security.AccessControlException", 
"message": "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\"" } }

.

So it migth be due to some proxy users. Usually "hdfs" is the super user/group so it should have all the permissions.

Can you please check who is owns the directory and what does the "hdfs-audit.log" say during that timestamp?

# su - hdfs -c "hdfs dfs -ls /ats"
# su - hdfs -c "hdfs dfs -ls /ats/done"

# less /var/log/hadoop/hdfs/hdfs-audit.log

.

Regarding your query who makes this call: It is ambari who invokes this script and makes the SETOWNER call on webhdfs

https://github.com/apache/ambari/blob/release-2.6.1/ambari-common/src/main/python/resource_managemen...

.

What might be the issue: So this issue can occur if you might have changed the "dfs.permissions.superusergroup" property value and removed the default "hdfs" group from there. So can you please check what is the value for this property in your "hdfs-site.xml" ?

[#3]

Hi Jay,

Thanks for your suggestions first!

The problem is "App Timeline Server / YARN" started failed due to "setowner" or "setpermission" failed.

1. I have checked that "hdfs" is superusergroup in "/etc/hadoop/2.5.3.0-37/0/hdfs-site.xml". I have reinstalled my environment with "2.5.3.0-37".

<property> <name>dfs.permissions.superusergroup</name> <value>hdfs</value> </property>

2. Permissions for /ats and /ats/done as below. I have tried to set owner to "yarn" but issue persists.

[root@centos-03 /]# su - hdfs -c "hdfs dfs -ls / |grep ats"

drwxr-xr-x - hdfs hadoop 0 2018-03-15 00:47 /ats

[root@centos-03 /]# su - hdfs -c "hdfs dfs -ls /ats |grep done"

drwxrwxrwx - hdfs hadoop 0 2018-03-14 21:48 /ats/done

3. Yes, the problem is from "hdfs_resource.py". This script will check and correct permission on the HDFS folders on Isilon (namenode and datanode). So here it tried to setowner.

  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 289, in action_delayed
    self._set_owner(self.target_status)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 436, in _set_owner
    self.util.run_command(self.main_resource.resource.target, 'SETOWNER', method='PUT', owner=owner, group=group, assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "AccessControlException", 
    "javaClassName": "org.apache.hadoop.security.AccessControlException", 
    "message": "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\""
  }
}

4. I tested above failed curl script and it seems user hdfs has no permission to set user "yarn".

---Failed with op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop. It was running required by hdfs_resource.py

[root@centos-03 /]# su - hdfs -c "curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop'"
{
   "RemoteException" : {
      "exception" : "AccessControlException",
      "javaClassName" : "org.apache.hadoop.security.AccessControlException",
      "message" : "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\""
   }
}
403

--- Succeed with SETOWNER&user.name=hdfs&owner=hdfs&group=hadoop

[root@centos-03 /]# su - hdfs -c "curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=hdfs&group=hadoop'"
200[root@centos-03 /]#

So the trick here is:

To start app timeline server, part of YARN, hdfs_resource.py will use hdfs to setowner and permission of user yarn. But it does NOT has such privilege to do that.

This is my understanding for this issue so far.