Hello, We just enabled FreeIPA integration in our Hortonworks cluster (HDP 2.5.3). We understand that part of the kerberization of Yarn, Ambari/yarn will try and delete the yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. The directories defined for these configs in our environment are actual mount points, so as expected Yarn throws this error message "OSError: [Errno 16] Device or resource busy: '/hadoop/yarn/local/01". Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 161, in <module>
Nodemanager().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 51, in start
self.configure(env) # FOR SECURITY
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 57, in configure
yarn(name="nodemanager")
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py", line 168, in yarn
action='delete'
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 114, in __new__
cls(names_list.pop(0), env, provider, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 208, in action_delete
sudo.rmtree(path)
File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 102, in rmtree
shutil.rmtree(path)
File "/usr/lib64/python2.7/shutil.py", line 256, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib64/python2.7/shutil.py", line 254, in rmtree
os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/hadoop/yarn/local/01' 2017-10-25 18:19:36,019 - checked_call returned (0, '')
2017-10-25 18:19:36,019 - Ensuring that hadoop has the correct symlink structure
2017-10-25 18:19:36,019 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-10-25 18:19:36,025 - Directory['/hadoop/yarn/local/01'] {'action': ['delete']}
2017-10-25 18:19:36,026 - Removing directory Directory['/hadoop/yarn/local/01'] and all its content
Command failed after 1 tries Any idea how do we get around this? Is there some script/flag we can modify for it to avoid attempting to delete these directories? Just an FYI, we can start the NM service on our DN manually. Any help would be appreciated. Thanks! K
... View more
