Why ambari host might have different public_host_name and host_name in OpenStack ?

[#1]

If we are working on some cloud environment like "OpenStack" then we might encounter an issue in which after installing the ambari agents we will see that some of the Quick Links in the ambari console is not having the expected hostname. For example if we navigate to Ambari --> HDFS --> "Resource Links" then we may see the incorrect hostnames. Actually those are detected as "public_host_name" by the ambari "/usr/lib/python2.6/site-packages/ambari_agent/hostname.py" script during every restart.

So in case of OpenStack it is possible that the "public_host_name" might be different from the "host_name" (`hostname -f`)

Here if we see that the Resource Link will show the links with the hostname "ambari1.example.com" (which is `hostname -f` but it is showing "ambari1.novalocal" hence the links are not working.

In such case we might find that the ambari "hosts" table in the ambari database has different values for the "public_host_name" and the "host_name"

select public_host_name, host_name from hosts
----------------------------------------------
public_host_name	   |	host_name
----------------------------------------------
ambari3.novalocal  |  ambari3.example.com
ambari2.novalocal  |  ambari2.example.com
ambari5.novalocal  |  ambari5.example.com
ambari1.novalocal  |  ambari1.example.com
ambari4.novalocal  |  ambari4.example.com

In this case we can refer to the following document to fix the hostname (or define a custom hostname)

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_ambari_reference_guide/content/_how_to_c...

.

.

** Permanently fix the public hostname: (Recommended)

1. Create a file with name : "/var/lib/ambari-agent/public_hostname.sh" then in that file add the following line:

#!/bin/sh
echo `hostname -f`

2. Make sure that the file "/var/lib/ambari-agent/public_hostname.sh" has proper execute permission. Example:

chmod 755 "/var/lib/ambari-agent/public_hostname.sh"

3. On every ambari-agent host edit the file "/etc/ambari-agent/conf/ambari-agent.ini" and in the [agent] section add the following line:

## Added following to customize the public hostname
public_hostname_script=/var/lib/ambari-agent/public_hostname.sh

NOTE: Users can also use the property "hostname_script" to customize the internal hostname.

3. Make sure that the changes are pushed to all the hosts present in the ambari cluster.

4. Now restart the agents.

ambari-agent restart

.

.

. Temporarily fix the public hostname using Ambari APIs:

Using Ambari APIs we can temporarily fix the host name (which will not survive the ambari agent restart).

1. List all the hostnames : https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts?fields=Hosts/public_host_name

Example output:

items: [{
    href: "https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts/ambari1.example.com",
    Hosts: {
    cluster_name: "ClusterDemo",
    host_name: "ambari1.example.com",
    public_host_name: "ambari1.novalocal"
    }
  },

Now lets change the public hostname "ambari1.novalocal" to "ambari1.example.com" :

- (A). Get the output of the URL: https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts/ambari1.example.com

- (B). Now from the output remove the very first line containing "href:" line

- (C). In the output file change the public_host_name to the desired name. and then run the following command:

curl --insecure -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d 
'{"Hosts" : {"public_host_name" : "ambari1.example.com"}}' 
https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts/ambari1.example.com

.

How ambari agent finds the hostname ?

Users can refer to the "/usr/lib/python2.6/site-packages/ambari_agent/hostname.py" script to know how ambari agents gets the "public_host_name" and "host_name".

Here we will see a simple python script "findHostname.py" to understand the logic which agent is using to determine the internal "host_name" and the "public_host_name"

import socket
import subprocess
import urllib2

cached_hostname = None
cached_public_hostname = None
try:
    print "\n####### finding 'hostname' using 'urllib2.urlopen' approach ########"
    handle = urllib2.urlopen('http://169.254.169.254/latest/meta-data/public-hostname', '', 2)
    str = handle.read()
    print 'str = ' + str
    handle.close()
    cached_public_hostname = str.lower()
    print '[Try] cached_public_hostname = ' + cached_public_hostname
    print("Read public hostname '" + cached_public_hostname + "' from http://169.254.169.254/latest/meta-data/public-hostname")

    print "\n\n####### finding 'hostname' using 'socket.getfqdn()' approach ########"
    cached_hostname = socket.getfqdn()
    print '[Try] cached_hostname = ' + cached_hostname
except:
    cached_public_hostname = socket.getfqdn().lower()
    print '[except] cached_public_hostname = ' + cached_public_hostname
    print("Read public hostname '" + cached_public_hostname + "' using socket.getfqdn()")

After running the above python script we may see the following output:

# python findHostname.py

####### finding 'hostname' using 'urllib2.urlopen' approach ########
str = ambari2.novalocal
[Try] cached_public_hostname = ambari2.novalocal
Read public hostname 'ambari2.novalocal' from http://169.254.169.254/latest/meta-data/public-hostname

####### finding 'hostname' using 'socket.getfqdn()' approach ########
[Try] cached_hostname = ambari2.example.com

[#2]

Brilliant article - both the answer and the explanation I was looking for - thanks for posting!

[#3]

Pin point explanation! Thanks Jay!

[#4]

Thank you very much for this!!!!, It solves the problem registering host using ambari 2.7.3 and hdp 3.1.4 awesome!