Member since
08-15-2019
2
Posts
0
Kudos Received
0
Solutions
03-15-2018
12:08 AM
Hi Jay, Thanks for your suggestions first! The problem is "App Timeline Server / YARN" started failed due to "setowner" or "setpermission" failed. 1. I have checked that "hdfs" is superusergroup in "/etc/hadoop/2.5.3.0-37/0/hdfs-site.xml". I have reinstalled my environment with "2.5.3.0-37". <property>
<name>dfs.permissions.superusergroup</name>
<value>hdfs</value>
</property> 2. Permissions for /ats and /ats/done as below. I have tried to set owner to "yarn" but issue persists. [root@centos-03 /]# su - hdfs -c "hdfs dfs -ls / |grep ats"
drwxr-xr-x - hdfs hadoop 0 2018-03-15 00:47 /ats [root@centos-03 /]# su - hdfs -c "hdfs dfs -ls /ats |grep done" drwxrwxrwx - hdfs hadoop 0 2018-03-14 21:48 /ats/done 3. Yes, the problem is from "hdfs_resource.py". This script will check and correct permission on the HDFS folders on Isilon (namenode and datanode). So here it tried to setowner. File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 289, in action_delayed
self._set_owner(self.target_status)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 436, in _set_owner
self.util.run_command(self.main_resource.resource.target, 'SETOWNER', method='PUT', owner=owner, group=group, assertable_result=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop'' returned status_code=403.
{
"RemoteException": {
"exception": "AccessControlException",
"javaClassName": "org.apache.hadoop.security.AccessControlException",
"message": "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\""
}
}
4. I tested above failed curl script and it seems user hdfs has no permission to set user "yarn".
---Failed with op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop. It was running required by hdfs_resource.py
[root@centos-03 /]# su - hdfs -c "curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop'"
{
"RemoteException" : {
"exception" : "AccessControlException",
"javaClassName" : "org.apache.hadoop.security.AccessControlException",
"message" : "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\""
}
}
403
--- Succeed with SETOWNER&user.name=hdfs&owner=hdfs&group=hadoop
[root@centos-03 /]# su - hdfs -c "curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=hdfs&group=hadoop'"
200[root@centos-03 /]#
So the trick here is: To start app timeline server, part of YARN, hdfs_resource.py will use hdfs to setowner and permission of user yarn. But it does NOT has such privilege to do that. This is my understanding for this issue so far.
... View more
03-10-2018
10:32 AM
Hi, I am working on deploy HDP 2.6.4.0 with Isilon 8.1.0.1. Now App Timeline Server and History Server start failed with similar errors: History Server: File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 436, in _set_owner
self.util.run_command(self.main_resource.resource.target, 'SETOWNER', method='PUT', owner=owner, group=group, assertable_result=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/tmp/entity-file-history/active?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop'' returned status_code=403.
{
"RemoteException": {
"exception": "AccessControlException",
"javaClassName": "org.apache.hadoop.security.AccessControlException",
"message": "Permission denied: user=hdfs, access=WRITE, path=\"/tmp/entity-file-history/active\""
}
} App Timeline Server: File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 436, in _set_owner
self.util.run_command(self.main_resource.resource.target, 'SETOWNER', method='PUT', owner=owner, group=group, assertable_result=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://h-hdfs.alex.isilon:8082/webhdfs/v1/ats/done?op=SETOWNER&user.name=hdfs&owner=yarn&group=hadoop'' returned status_code=403.
{
"RemoteException": {
"exception": "AccessControlException",
"javaClassName": "org.apache.hadoop.security.AccessControlException",
"message": "Permission denied: user=hdfs, access=WRITE, path=\"/ats/done\""
}
} I am not sure which user is used here to run "SETOWNER" action defined in "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py". Is there anybody meet such this issue? Thank you!
... View more
Labels: