hostname and canonical name for this host are not consistent when checked from a Java process

[#1]

i have run this command on all hosts

# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest

and get this as a output :

[root@instance-1 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.4", "hostname": "instance-1", "canonicalname": "instance-1.c.sacred-evening-197206.internal", "localhostDuration": "4", "canonicalnameDuration": "0" }

[root@instance-2 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.5", "hostname": "instance-2", "canonicalname": "instance-2.c.sacred-evening-197206.internal", "localhostDuration": "5", "canonicalnameDuration": "0" }

[root@instance-3 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.c loudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.2", "hostname": "instance-3", "canonicalname": " instance-3.c.sacred-evening-197206.internal", "localhostDuration": "4", "canonic alnameDuration": "0" }

but in my /etc/sysconfig/network and /etc/hosts file i have mentioned FQDN then why its showing instance-1, instance-2, instance-3 while running above command.

please suggest

[#2]

Can you post you /etc/hosts file?

[#3]

[root@instance-1 Admin]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

10.142.0.5 instance-2.c.sacred-evening-197206.internal instance-2
10.142.0.2 instance-3.c.sacred-evening-197206.internal instance-3

10.142.0.4 instance-1.c.sacred-evening-197206.internal instance-1 # Added by Google
169.254.169.254 metadata.google.internal # Added by Google
[root@instance-1 Admin]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=instance-1.c.sacred-evening-197206.internal

[root@instance-1 Admin]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search c.sacred-evening-197206.internal. google.internal.
nameserver 169.254.169.254

this file resolv.conf is not fetching the excat FQDN. 

What to do please suggest.

[#4]

It is strange. Your config, seems to be ok. I don't know what is the problem.

I could recommend check the output of

hostname -f

and

host -t A 10.142.0.4

if the output is the FQDN, then probably something is wrong with your version of Dnstest.

Can you also check the "/etc/nsswitch.conf" file?

Usually the hosts line is:

hosts:          files dns

Can you check if something else is before the above? E.g. "sss files dns".

The order matters. If "files" is first, then the local "/etc/hosts" will be checked first http://man7.org/linux/man-pages/man5/nsswitch.conf.5.html

[#5]

[root@instance-1 Admin]# hostname -f
instance-1.c.sacred-evening-197206.internal
[root@instance-1 Admin]# host -t A 10.142.0.4
4.0.142.10.in-addr.arpa domain name pointer instance-1.c.sacred-evening-197206.internal.
[root@instance-1 Admin]# cat /etc/nsswitch.conf
#
# /etc/nsswitch.conf
#
# An example Name Service Switch config file. This file should be
# sorted with the most-used services at the beginning.
#
# The entry '[NOTFOUND=return]' means that the search for an
# entry should stop if the search in the previous entry turned
# up nothing. Note that if the search failed due to some other reason
# (like no NIS server responding) then the search continues with the
# next entry.
#
# Valid entries include:
#
# nisplus Use NIS+ (NIS version 3)
# nis Use NIS (NIS version 2), also called YP
# dns Use DNS (Domain Name Service)
# files Use the local files
# db Use the local database (.db) files
# compat Use NIS on compat mode
# hesiod Use Hesiod for user lookups
# [NOTFOUND=return] Stop searching if not found so far
#

# To use db, put the "db" in front of "files" for entries you want to be
# looked up first in the databases
#
# Example:
#passwd: db files nisplus nis
#shadow: db files nisplus nis
#group: db files nisplus nis

passwd: files
shadow: files
group: files

#hosts: db files nisplus nis dns
hosts: files dns

# Example - obey only what nisplus tells us...
#services: nisplus [NOTFOUND=return] files
#networks: nisplus [NOTFOUND=return] files
#protocols: nisplus [NOTFOUND=return] files
#rpc: nisplus [NOTFOUND=return] files
#ethers: nisplus [NOTFOUND=return] files
#netmasks: nisplus [NOTFOUND=return] files

bootparams: nisplus [NOTFOUND=return] files

ethers: files
netmasks: files
networks: files
protocols: files
rpc: files
services: files

netgroup: nisplus

publickey: nisplus

automount: files nisplus
aliases: files nisplus

[root@instance-1 Admin]#

i guess the squence is right. And hostname -f and hostname -t A 10.142.04 gives FQDN of hostname.

is something wrong with my version of Dnstest ot its something else?

What can I do next

kindly suggest.

[#6]

Everything seem to be ok. I have the same configuration and cannot reproduce your issue.

I have CM 5.14.3 installed so my jar is /usr/share/cmf/lib/agent-5.14.3.jar

What version are you using?

[#7]

hello @GeKas

I am using Version: Cloudera Enterprise Trial 5.14.3

I am also having same jar on the same path.

Is anything else where we can look to fix the issues. Beacuse of mismatch in hostname and conical name, health of cluser is in bad condition and all services are marked as red.

Thanks for your valuable time and support !!

[#8]

Same issue. Were you able to fix this?

[#9]

@sim6,

The DNSTest class does simple tests like this to get your "hostname":

InetAddress addr = InetAddress.getLocalHost();

String hostname = addr.getHostName();

I believe that Java calls 'uname' under the hood to get the host name.

I believe that 'uname' gets the hostname from the proc location:

/proc/sys/kernel/hostname

Anyway, try checking

/etc/sysconfig/network

/etc/hosts

Make sure the FQDN is specified

Check "uname -a"

Check "cat /proc/sys/kernel/hostname"

[#10]

uname -a returns NON-FQDN hostname. Can I edit 

/proc/sys/kernel/hostname

and change the value to FQDN hostname?

At the same time, I have 2 servers for which it's alright and does not change after 24 hours of restart (I guess I did not mention I start getting the different names after 24 hours of server restart , turns out it happens on network restart, something we don't do but GCP does). That one also has NON-FQDN name in 

/proc/sys/kernel/hostname and there it however works fine