The hostname and canonical name for this host are not consistent

[#1]

Hi All,

When I added a new node in cloudera cluster, The node went to bad health and shows error message like "The hostname and canonical name for this host are not consistent when checked from a Java process" Can any one help us to fix this issue.

Thanks,

Sathishkumar M

[#2 · Accepted solution]

If you're running Red Hat or CenOS, ensure the /etc/sysconfig/network file
has the HOSTNAME field with the FQDN, not just the host name.

HOSTNAME=server1.example.com (good)
HOSTNAME=server1 (bad)

Edit this and reboot the host. Just restarting network services might not
work. Then let us know if the error still continues to be logged.

[#3 · Accepted solution]

If you're running Red Hat or CenOS, ensure the /etc/sysconfig/network file
has the HOSTNAME field with the FQDN, not just the host name.

HOSTNAME=server1.example.com (good)
HOSTNAME=server1 (bad)

Edit this and reboot the host. Just restarting network services might not
work. Then let us know if the error still continues to be logged.

[#4]

Thanks Gautam, It  is working. 

But, Other nodes in the same cluster dont have this FQDN in that path /etc/sysconfig/network and those are working fine.Is there any specific reason?

[#5 · Accepted solution]

That is hard to say. Please run this python one-liner on a host where you
don't get complaints and on the host you just fixed. Do they look similar?

# python -c "import socket; print socket.getfqdn(); print
socket.gethostbyname(socket.getfqdn())"

[#6]

Yes,The onliner diplaying the FQDN in both the nodes.  But we tried after removing that FQDN in /etc/sysconfig/network file also and we got the same output.

[#7]

Just editing the entry won't suffice. Changes to that file need a reboot to
take effect.

[#8]

Thanks Gautam for your reply. We checked only after the reboot.

[#9]

From what I understand, you have some hosts with HOSTNAME=server1 and some with HOSTNAME=server1.example.com. CM does not complain about all hosts which have HOSTNAME=server1. Is this right?

Are you able to provide the python script output from both types of hosts? What OS are these hosts running?

[#10]

Network file for that node which is in bad health before change 

cat /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=node10

Output of Python code

python -c "import socket; print socket.getfqdn(); print
socket.gethostbyname(socket.getfqdn())"

node10.cluster1.com

Network file for that node which is already in good health

cat /etc/sysconfig/network

python -c "import socket; print socket.getfqdn(); print
socket.gethostbyname(socket.getfqdn())"

node6.cluster1.com

Thanks

Sathishkumar M

[#11]

Satish, I am failing to see the problem here. Maybe I'm missing something obvious?

You were getting warnings for node10 earlier but don't any more. Does node6 have "HOSTNAME=node6" and still doesn't generate warnings?

[#12]

The python command should be all one line, or use a "\" to continue on a new line.

For example:

$ python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"

or
$ python -c "import socket; print socket.getfqdn(); \
print socket.gethostbyname(socket.getfqdn())"

[#13]

Hi ,  I also had the same issue in RHEL 7 . As suggested I modified the "HOSTNAME" attribute in /etc/sysconfig/network file to FQDN. Following that I also modified the hostname to FQDN using the following command:

hostnamectl set-hostname "<your hostname FQDN>" --static 

Followed by this  I restarted cloudera-scm-agent . This worked for me , there were no more critical warnings on hostname and canonical hostname .

[#14]

What am I doing wrong? How can I correct this?
1. DNS Resolution Error
2. Job History Server Web UI going to

"This site can’t be reached
instance-1.c.1345678.internal’s server IP address could not be found."


hostname -f
instance-1.c.12345678.internal

cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.123.0.2 instance-1.12345678.internal instance-1 # Added by Google
123.456.789.012 metadata.google.internal # Added by Google

cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost

python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())";
instance-1.c.12345678.internal
10.123.0.2

ifconfig eth0 | grep "inet addr"
inet addr:10.123.0.2 Bcast:10.123.0.2 Mask:255.255.255.255

[#15]

@Genthandsome,

Hi,

I see you are posting on a lot of different old threads.  In the future, please create a new thread and explain the issue you are seeing like you did here.  Thank you.

If you google "server IP address could not be found" you will find information that helps point you in the right direction about what that means.  You are getting that in Chrome, I presume, so I also guess that you are attempting to connect to the Job History Server UI from a browser that is on a different host than nstance-1.c.12345678.internal

That means your local host (where your browser is installed) cannot resolve the hostname "nstance-1.c.12345678.internal".  You will need to ensure your client host has hosts file or DNS configured to allow it to resolve the host to IP address.

[#16]

error still persists even after restarting

am using centOs 6

[root@instance-1 Admin]# hostname -f
instance-1.c.sacred-evening-197206.internal
[root@instance-1 Admin]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=instance-1.c.sacred-evening-197206.internal
[root@instance-1 Admin]#

what to do kindly suggest.

[#17]

hi @prabhat10,

Since this is an old thread, we should make sure you are indeed seeing the same issue that is described.

Please let us know what you are doing when you see the problem and what the problem is.

If you are seeing the message that is in the initial comment of this thread, then to get some more insight, I recommend running the DNSTest manually like this on a cluster host:

(make sure java is in your path or specify the full path to the java file)

# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest

The output will be a JSON format file like this:

{"status": "0", "ip": "122.168.100.211", "hostname": "host.example.com", "canonicalname": "host.example.com", "localhostDuration": "4", "canonicalnameDuration": "0" }

This might shed some light on what is going on.  I'd check your hosts file or DNS depending on how your host resolution is configured on that host.

Regards,

Ben

[#18]

hello@bgooley

i have run this command 

# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest

and get this as a output :

[root@instance-1 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.4", "hostname": "instance-1", "canonicalname": "instance-1.c.sacred-evening-197206.internal", "localhostDuration": "4", "canonicalnameDuration": "0" }
[root@instance-1 Admin]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=instance-1.c.sacred-evening-197206.internal

[root@instance-1 Admin]# java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
[root@instance-1 Admin]# javac -version
bash: javac: command not found

do i need to run the above command on each hosts and other hosts java is not installed, do i need to instaal it manually. On doing java -version its showing command not found.

[root@instance-1 jdk1.7.0_67-cloudera]# alternatives --display java| grep Current
Current `best' version is /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java.
[root@instance-1 jdk1.7.0_67-cloudera]# ls /usr/java/
jdk1.7.0_67-cloudera
[root@instance-1 jdk1.7.0_67-cloudera]# ls /usr/java/jdk1.7.0_67-cloudera/
bin lib src.zip
COPYRIGHT LICENSE THIRDPARTYLICENSEREADME-JAVAFX.txt
db man THIRDPARTYLICENSEREADME.txt
include README.html
jre release

kindly suggest!!

[#19]

i have run this command on all hosts

# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest

and get this as a output :

[root@instance-1 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.4", "hostname": "instance-1", "canonicalname": "instance-1.c.sacred-evening-197206.internal", "localhostDuration": "4", "canonicalnameDuration": "0" }

[root@instance-2 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.5", "hostname": "instance-2", "canonicalname": "instance-2.c.sacred-evening-197206.internal", "localhostDuration": "5", "canonicalnameDuration": "0" }

[root@instance-3 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.c loudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.2", "hostname": "instance-3", "canonicalname": " instance-3.c.sacred-evening-197206.internal", "localhostDuration": "4", "canonic alnameDuration": "0" }

 but in my /etc/sysconfig/network and /etc/hosts file i have mentioned FQDN then why its showing instance-1, instance-2, instance-3 while running above command.

What to do next kindly suggest.

[#20]

@prabhat10, I'm sorry that I couldn't reply sooner.

The DnsTest command is what Cloudera Manager runs to check your host and canonical host names.

The problem is that your hostname and canonical names are not the same:

"hostname": "instance-1",

"canonicalname": "instance-1.c.sacred-evening-197206.internal

My guess is this has to do with your /etc/hosts file or general DNS resolution

Have you defined the host in /etc/hosts?

If so, make sure it is in the following format:

IP  FQDN

The FQDN name must be first

[#21]

i have run this command on all hosts

# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest

and get this as a output :

[root@instance-1 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.4", "hostname": "instance-1", "canonicalname": "instance-1.c.sacred-evening-197206.internal", "localhostDuration": "4", "canonicalnameDuration": "0" }

[root@instance-2 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.5", "hostname": "instance-2", "canonicalname": "instance-2.c.sacred-evening-197206.internal", "localhostDuration": "5", "canonicalnameDuration": "0" }

[root@instance-3 Admin]# java -classpath /usr/share/cmf/lib/agent-5.*.jar com.c loudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.142.0.2", "hostname": "instance-3", "canonicalname": " instance-3.c.sacred-evening-197206.internal", "localhostDuration": "4", "canonic alnameDuration": "0" }

but in my /etc/sysconfig/network and /etc/hosts file i have mentioned FQDN then why its showing instance-1, instance-2, instance-3 while running above command.

What to do next?

Kindly suggest

[#22]

I am having same problem here.

I used to install Cloudera Manager on Centos 7 with no issues.

But in Centos 6(since impala is not supported on Centos 7, had to try Centos 6), getting this error on all hosts. 

My /etc/sysconfig/network file is like this:

$ cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=cixa.c.gib-data.internal

I rebooted all hosts after setting HOSTNAME in /etc/sysconfig/network file.

All other nodes share same configuration. 

Host inspection gives no warning.

In host status, this error message is shown:

"The hostname and canonical name for this host are not consistent when checked from a Java process"

$ python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"
cixa.c.gib-data.internal
10.128.0.2

All hosts answer correctly to nslookup and host commands.

All hosts can shh and ping to all others with FQDN.