Host not found, hostname=

[#1]

At wits end wit this hortwork work platform , trying to get it kerborized.

The phase where it Testing Kerberos clients, keep getting this error that it cannot find the ambari host and displays the fqdn

The ambari host is name resolved by 2 hosted domains internally with no problem. ie ambari.example.com and ambari.corp.example.com. I also added this in /etc/hosts files on each server in cluster including ambari server , each maser node , slave node and DC on our network to eliminate DNS as an issue ( which I cannot believe it is as at teh command line these things can resolve all day long to each other) What logs can I examine to give more information on this error please?

var/log/ambari-server is not much help

[#2]

@Tom Burke

If you define two hostname s inside the "/etc/hosts" file then you might find issues in kerberized environment

Example:

192.xx.yy.170    ambari.example.com   ambari.corp.example.com

.

In thew above case we might see that the kerberos clients resolves the incorrect hsotname. So can you please keep only one host there and then try again.

Also can you please share the complete message where you see "Host not found, hostname" that way we can check the exact code base to understand in which conditions it might show that message

[#3]

Hi Jay , thanks for reply.

I changed /etc/hosts on ambari to reflect your suggestion ( I have not done all in cluster as yet)

10.x.x.x ambari.example.com

trie to enable , still returns error on creating principles for ambari ....the error from /var/log/ambari-server/ambari-server.log is below. ( is there anyplace else to look?)

	2019-02-05 02:23:07,223  WARN [Server Action Executor Worker 1760] ServerActionExecutor:471 - Task #1760 failed to complete execution due to thrown exception: org.apache.ambari.server.HostNotFoundException:Host not found, hostname=ambari.example.com
org.apache.ambari.server.HostNotFoundException: Host not found, hostname=ambari.example.com
2019-02-05 19:29:02,615  WARN [Server Action Executor Worker 1801] ServerActionExecutor:471 - Task #1801 failed to complete execution due to thrown exception: org.apache.ambari.server.HostNotFoundException:Host not found, hostname=ambari.example.com
org.apache.ambari.server.HostNotFoundException: Host not found, hostname=ambari.example.com

[#4]

Does the Amabri server have to be "in" the cluster?

[#5]

	2019-02-05 02:23:07,223  WARN [Server Action Executor Worker 1760] ServerActionExecutor:471 - Task #1760 failed to complete execution due to thrown exception: org.apache.ambari.server.HostNotFoundException:Host not found, hostname=ambari.example.com
org.apache.ambari.server.HostNotFoundException: Host not found, hostname=ambari.example.com
2019-02-05 19:29:02,615  WARN [Server Action Executor Worker 1801] ServerActionExecutor:471 - Task #1801 failed to complete execution due to thrown exception: org.apache.ambari.server.HostNotFoundException:Host not found, hostname=ambari.example.com
org.apache.ambari.server.HostNotFoundException: Host not found, hostname=ambari.example.com

[#6]

Hi Jay thanks. Above is the error log from /var/log/ambari-server/ambari-server.log I added removed the corp/example/.cpom entries from /etc/hosts

Aside questions of anyone can answer...

: Is there any documentation to suggest that you cannot use ubuntu 18.04 as the server? every doc I read does not reference this version, seems to stop at ubuntu 12 or 14...

Does the ambari server need to be host member of cluster? ...

[#7]

@Tom Burke

Regarding Ubuntu-18 supportability... you can refer to : Support matrix and click on your version of Ambari to see which all OS it is tested and certified to work with.

https://supportmatrix.hortonworks.com/

.

Regarding other query "Does the ambari server need to be host member of cluster? ..."

No, it is not mandatory to make AmbariServer host to be a member of Cluster.

[#8]

@Tom Burke

Ideally the "/etc/hosts" file kept identical on all cluster nodes to avoid any kind of hostname confusions.

.

It is not mandatory to install Agent on the AmbariServer host. (that is optional)

.

Regarding the "HostNotFoundException" can you please share the output of the following API call to see the hostss (to find out if any conflict)

# curl -H "X-Requested-By: ambari" -u admin:admin http://$AMBARI_HOSTNAME:8080/api/v1/clusters/$CLUSTER_NAME/hosts?fields=Hosts/ip,Hosts/host_name

.

[#9]

ambari using ssl 8443 so used this otherwise connection refused.

root@ambari:~# curl -H "X-Requested-By: ambari" -u admin:admin http://ambari.example.com:8443/api/v1/clusters/cluster-name/hosts?fields=Hosts/ip,Hosts/host_name

Warning: Binary output can mess up your terminal. Use "--output -" to tell

Warning: curl to output it to your terminal anyway, or consider "--output

Warning: <FILE>" to save to a file.

[#10]

ambari using ssl 8443 so used this otherwise connection refused.

root@ambari:~# curl -H "X-Requested-By: ambari" -u admin:admin http://ambari.example.com:8443/api/v1/clusters/cluster-name/hosts?fields=Hosts/ip,Hosts/host_name

Warning: Binary output can mess up your terminal. Use "--output -" to tell

Warning: curl to output it to your terminal anyway, or consider "--output

Warning: <FILE>" to save to a file.

[#11]

@Tom Burke

You can also open the same URL from the Browser where you have logged in the Ambari UI as well.

http://ambari.example.com:8443/api/v1/clusters/cluster-name/hosts?fields=Hosts/ip,Hosts/host_name

.

(OR) You can use the following curl call to produce the JSON output to some file like "/tmp/hosts.json". Also pleas emake sure to use the correct cluster name in the same URL.

# curl -k -H "X-Requested-By: ambari" -u admin:admin "http://ambari.example.com:8443/api/v1/clusters/cluster-name/hosts?fields=Hosts/ip,Hosts/host_name" -o "/tmp/hosts.json"

.

[#12]

hi @Jay Kumar SenSharma,

i am upgrading from HDP 2.6.2 to HDP 3.1, i am using Ambari 2.7.3.0 and currently stuck on (generate missing principal ), my cluster is kerberized, the machine hosting ambari server doesn't have ambari agent, and i am also getting the same error (host not found) for the ambari master machine , below is the exact error from ambari server log

2019-05-07 16:55:58,023  WARN [Server Action Executor Worker 3611] ServerActionExecutor:471 - Task #3611 failed to complete execution due to thrown exception: org.apache.ambari.server.HostNotFoundException:Host not found, hostname=MYHOSTNAME.com
org.apache.ambari.server.HostNotFoundException: Host not found, hostname=MYHOSTNAME.com
        at org.apache.ambari.server.state.cluster.ClustersImpl.getHost(ClustersImpl.java:456)
        at org.apache.ambari.server.state.ConfigHelper.getEffectiveDesiredTags(ConfigHelper.java:189)
        at org.apache.ambari.server.state.ConfigHelper.getEffectiveDesiredTags(ConfigHelper.java:173)
        at org.apache.ambari.server.controller.AmbariManagementControllerImpl.findConfigurationTagsWithOverrides(AmbariManagementControllerImpl.java:2371)
        at sun.reflect.GeneratedMethodAccessor424.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
        at com.sun.proxy.$Proxy131.findConfigurationTagsWithOverrides(Unknown Source)
        at org.apache.ambari.server.state.ConfigHelper.calculateExistingConfigurations(ConfigHelper.java:2158)
        at org.apache.ambari.server.controller.KerberosHelperImpl.calculateConfigurations(KerberosHelperImpl.java:1725)
        at org.apache.ambari.server.controller.KerberosHelperImpl.getActiveIdentities(KerberosHelperImpl.java:1800)
        at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.calculateServiceIdentities(KerberosServerAction.java:511)
        at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:455)
        at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.execute(CreatePrincipalsServerAction.java:92)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466)
        at java.lang.Thread.run(Thread.java:745)

can you please suggest what to do?

[#13]

@jsensharma This issue has been observed in the internal lab cluster as well.

curl -H "X-Requested-By: ambari" -u admin:pnarayanasamy"http://c2237-node1.labs.support.hortonworks.com:8080/api/v1/clusters/c2237/hosts"

ambari server & kdc host c2237-node1.labs.support.hortonworks.com is not listed in the items array response.

/etc/hosts having only one fqdn for all the ip's.

[#14]

Hi all,

I had faced the same issue while kerberizing a HDP 3.1.0 cluster integrated with Iislon.

The Ambari serer is a built on a Postgres DB cluster, hence it was NOT in the hadoop cluster.

so it was NOT on the host list.

[root@hdpdb ~]# curl -H "X-Requested-By: ambari" -u admin:admin "http://hdpdb.gz.local:8080/api/v1/clusters/Panyu/hosts"

I had to add the ambari server from adding host wizard and just install the clients.

It worked around the "Host not found, hostname= xxxx" issue in my case. 

Hope it will help, cheer!