We add and remove spot instances on demand from our cluster. We use the internal DNS as the hostname (ex "ip-10-0-3-50.ec2.internal") and AWS hostid as CM hostid (ex i-56ab56c5). We attempt to remove hosts and roles from the cluster when we remove spot nodes or when spot nodes are taken away from us. We have noticed very odd behavior after upgrading from CDH 5.4.0 to CDH 5.7.2:
1. api.get_all_hosts() cluster.list_hosts() do not return all hosts if the hostname is identical even though the host-id is not the same. Cloudera Manager UI displays the two hosts separately as it should, since the uniqueness is based on host-id and not the hostname.
2. When you attempt to delete a host by calling hosts.delete_host() and pass the host-id to delete, it sometimes attempts to remove the host that has the identical hostname with a different host id and generates a message such as:
Traceback (most recent call last): File "/Users/alexrovner/git/magnetic_hadoop/clusterk/deco.py", line 62, in run delete_host(api, h) File "/Users/alexrovner/.virtualenvs/magnetic-py2.7/lib/python2.7/site-packages/cm_api/endpoints/hosts.py", line 63, in delete_host return call(resource_root.delete, "%s/%s" % (HOSTS_PATH, host_id), ApiHost) File "/Users/alexrovner/.virtualenvs/magnetic-py2.7/lib/python2.7/site-packages/cm_api/endpoints/types.py", line 139, in call ret = method(path, params=params) File "/Users/alexrovner/.virtualenvs/magnetic-py2.7/lib/python2.7/site-packages/cm_api/resource.py", line 134, in delete return self.invoke("DELETE", relpath, params) File "/Users/alexrovner/.virtualenvs/magnetic-py2.7/lib/python2.7/site-packages/cm_api/resource.py", line 73, in invoke headers=headers) File "/Users/alexrovner/.virtualenvs/magnetic-py2.7/lib/python2.7/site-packages/cm_api/http_client.py", line 174, in execute raise self._exc_class(ex) ApiException: Cannot delete host with associated roles. (error 400)
This is a major issue for us, so any help here would be appreciated .