Getting could not find hostname or IP address of SCM server error during CDH installation

[#1]

Hello ,

I've installed Cloudera manager 4.8.1 . I am trying to install the CDH and other components where in i get the following error message
"Installation failed. Failed to execute installation script." . When i Click on "Details" link , able to see below error details . Could you please assit me on this.

Not sure i miss some basic information. I am a beginner.:)

/tmp/scm_prepare_node.rtuY9525
/tmp/scm_prepare_node.rtuY9525/scm_prepare_node.sh: could not find hostname or IP address of SCM server
usage: /tmp/scm_prepare_node.rtuY9525/scm_prepare_node.sh [options]

Prepares this node for use by Cloudera Service and Configuration Manager (SCM):
o Installs necessary SCM agent packages.
o Installs necessary CDH packages.
o Configures the SCM agent.
o Starts the SCM agent.

[#2]

Hello Venkat,

"could not find hostname or IP address of SCM server" would normally mean that the host's name is not set up correctly.

Run the following commands and let me know what it says:

# hostname -f

#cat /etc/hosts

# python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"

Regards
Gautam

[#3]

Thanks Gautam to look into the issue.

I use two machines for clustering. Cloud era manager is installed on the below machine and i ran the command you provided.Below is the result

----------------------------

[root@ofss2311682 hadoop]# hostname -f
ofss2311682.in.oracle.com
[root@ofss2311682 hadoop]# cat /etc/hosts
# HEADER: This file was autogenerated at Fri Aug 22 10:59:22 +0000 2014
# HEADER: by puppet. While it can still be managed manually, it
# HEADER: is definitely not recommended.
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1 localhost.localdomain loghost localhost
10.184.153.71 ofss2311682.in.oracle.com ofss2311682
#127.0.0.1 localhost.localdomain loghost localhost
127.0.0.1 localhost.localdomain loghost localhost
[root@ofss2311682 hadoop]# python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"
ofss2311682.in.oracle.com
10.184.153.71
[root@ofss2311682 hadoop]#

-------------------------------

Second Machine - 

[root@ofss220385 etc]# hostname -f
ofss220385.in.oracle.com
[root@ofss220385 etc]# cat /etc/hosts
# HEADER: This file was autogenerated at Fri Aug 22 10:36:34 +0000 2014
# HEADER: by puppet. While it can still be managed manually, it
# HEADER: is definitely not recommended.
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1 localhost.localdomain loghost localhost
#10.184.135.19 localhost.in.oracle.com localhost
10.184.135.19 ofss220385.in.oracle.com ofss220385
#127.0.0.1 localhost.localdomain loghost localhost
127.0.0.1 localhost.localdomain loghost localhost
[root@ofss220385 etc]# python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"
ofss220385.in.oracle.com
10.184.135.19
[root@ofss220385 etc]#

[#4]

cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.123.0.4 instance-1.c.grand-master-123456.internal instance-1 # Added by Google
345.678.901.234 metadata.google.internal # Added by Google


cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost

python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())";
instance-1.c.grand-master-123456.internal
10.123.0.4

what do I need to do?

[#5]

@Genthandsome,

We are happy to help, but need to know more.

What steps have you taken and what problem do you observe?   Please provide logs or screen shots that show how you detected the problem.

[#6]

This bug actually still exists, it happens when the script reads the hosts in the wrong order. If you retry over and over, eventually the script will flip the two values and install:

Detecting Cloudera Manager Server...
BEGIN host -t PTR xxx.xx.xx.xxx
1xx.xx.xx.xxx.in-addr.arpa domain name pointer cdh-mgr.mydomain.corp  <--- this is write
1xx.xx.xx.xxx.in-addr.arpa domain name pointer localhost.localdomain.  <--- this is wrong
END (0) 
using localhost.localdomain as scm server hostname   <--- this picked up the last entry 
BEGIN which python 
/usr/bin/python

It should look like:

Detecting Cloudera Manager Server...
BEGIN host -t PTR xxx.xx.xx.xxx
1xx.xx.xx.xxx.in-addr.arpa domain name pointer localhost.localdomain.  <--- this is wrong
1xx.xx.xx.xxx.in-addr.arpa domain name pointer cdh-mgr.mydomain.corp  <--- this is write
END (0) 
using cdh-mgr.mydomain.corp as scm server hostname   <--- this picked up the last entry 
BEGIN which python 
/usr/bin/python

[#7]

Hello @jpetro416 ,

thank you for reporting the behavior that CM installation fails when it picks up a wrong hostname.

In the log segment you've shared it seems to be there is a dot after localhost.localdomain:

localhost.localdomain.

To alleviate the issue once hit it, please:

1. update the /etc/hosts file with the correct hostnames
2. copy this corrected file to all hosts
3. update /etc/cloudera-scm-agent/config.ini to point to the CM server hostname
4. hard restart the agent:
   # sudo systemctl stop supervisord
   # sudo systemctl start cloudera-scm-agent
5. add the hosts to the cluster
6. install Management Services

Steps were verified for CM6.2.

Please let me know if I've addressed your observation.

Thank you:
Ferenc