Adding a new host via the Ambari results in a "org.apache.ambari.server.HostNotFoundException: Host not found" exception

[#1]

SYMPTOM:

While adding a new host via the Ambari , results in an exception as below:

#####
11 Apr 2016 10:52:24,629 ERROR [qtp-client-81] AbstractResourceProvider:279 - Caught AmbariException when creating a res 
ource 
org.apache.ambari.server.HostNotFoundException: Host not found, hostname=hostname_123.abc.xyz.com 
at org.apache.ambari.server.state.cluster.ClustersImpl.getHost(ClustersImpl.java:343) 
at org.apache.ambari.server.state.ConfigHelper.getEffectiveDesiredTags(ConfigHelper.java:108) 
at org.apache.ambari.server.controller.AmbariManagementControllerImpl.findConfigurationTagsWithOverrides(AmbariM 
anagementControllerImpl.java:1820) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) 
at com.sun.proxy.$Proxy82.findConfigurationTagsWithOverrides(Unknown Source) 
at org.apache.ambari.server.controller.AmbariActionExecutionHelper.addExecutionCommandsToStage(AmbariActionExecu 
tionHelper.java:372) 
at org.apache.ambari.server.controller.AmbariManagementControllerImpl.createAction(AmbariManagementControllerImp 
l.java:3366) 
at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:16 
5) 
at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:16 
2) 
at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvide 
r.java:272) 
at org.apache.ambari.server.controller.internal.RequestResourceProvider.createResources(RequestResourceProvider. 
java:162) 
at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java 
:289) 
at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:7 
6) 
at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36) 
at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72) 
at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135) 
at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:105) 
at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:74) 
at org.apache.ambari.server.api.services.RequestService.createRequests(RequestService.java:145) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

###########

ROOT CAUSE:

The issue usually happens when there is a conflicting entry in the /etc/hosts for the node which we are trying to add. This could be due to the fact that the hostname entry in the /etc/hosts file could be "incorrectly" resolving out to an "incorrect" IP address or vise versa. The Ambari agent uses the script "/usr/lib/python2.6/site-packages/ambari_agent/hostname.py" to push the updates to the ambari DB / server. The script specifically calls out / updates the hostname based on the code below:

####### 

try: 
scriptname = config.get('agent', 'hostname_script') 
try: 
osStat = subprocess.Popen([scriptname], stdout=subprocess.PIPE, stderr=subprocess.PIPE) 
out, err = osStat.communicate() 
if (0 == osStat.returncode and 0 != len(out.strip())): 
cached_hostname = out.strip() 
else: 
cached_hostname = socket.getfqdn() 
except: 
cached_hostname = socket.getfqdn() 
except: 
cached_hostname = socket.getfqdn() 
cached_hostname = cached_hostname.lower() 
return cached_hostname 

##### 

Here "socket.getfqdn()" will always look up for the /etc/hosts and update the "cached_hostname" which is then pushed out to the Ambari DB here.

As a simple check we can do a following check from the host itself which we have trouble adding to determine if the "socket.getfqd()" results in the right hostname as done below:

####
[root@sandbox ~]# python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> print socket.getfqdn()
sandbox.hortonworks.com
>>>
#####

The output printed by the "print socket.getfqdn()" should match to the entry that is captured in the /etc/hosts file.

SOLUTION:

Update the /etc/hosts entry to fix the incorrect entry and add the host back through Ambari again which then installs the Ambari agent back and updates the right entry in the Ambari DB