Hi Jay, Thanks for your suggestions first! The problem is "App Timeline Server / YARN" started failed due to "setowner" or "setpermission" failed. 1. I have checked that "hdfs" is superusergroup in "/etc/hadoop/2.5.3.0-37/0/hdfs-site.xml". I have reinstalled my environment with "2.5.3.0-37". <property> <name>dfs.permissions.superusergroup</name> <value>hdfs</value> </property> 2. Permissions for /ats and /ats/done as below. I have tried to set owner to "yarn" but issue persists. [root@centos-03 /]# su - hdfs -c "hdfs dfs -ls / |grep ats" drwxr-xr-x - hdfs hadoop 0 2018-03-15 00:47 /ats [root@centos-03 /]# su - hdfs -c "hdfs dfs -ls /ats |grep done" drwxrwxrwx - hdfs hadoop 0 2018-03-14 21:48 /ats/done 3. Yes, the problem is from "hdfs_resource.py". This script will check and correct permission on the HDFS folders on Isilon (namenode and datanode). So here it tried to setowner. File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 289, in action_delayed self._set_owner(self.target_status) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 436, in _set_owner self.util.run_command(self.main_resource.resource.target, 'SETOWNER', method='PUT', owner=owner, group=group, assertable_result=False) File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop'' returned status_code=403. { "RemoteException": { "exception": "AccessControlException", "javaClassName": "org.apache.hadoop.security.AccessControlException", "message": "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\"" } } 4. I tested above failed curl script and it seems user hdfs has no permission to set user "yarn". ---Failed with op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop. It was running required by hdfs_resource.py [root@centos-03 /]# su - hdfs -c "curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop'" { "RemoteException" : { "exception" : "AccessControlException", "javaClassName" : "org.apache.hadoop.security.AccessControlException", "message" : "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\"" } } 403 --- Succeed with SETOWNER&user.name=hdfs&owner=hdfs&group=hadoop [root@centos-03 /]# su - hdfs -c "curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=hdfs&group=hadoop'" 200[root@centos-03 /]# So the trick here is: To start app timeline server, part of YARN, hdfs_resource.py will use hdfs to setowner and permission of user yarn. But it does NOT has such privilege to do that. This is my understanding for this issue so far. ... View more