Brand new install, [Errno 99] Cannot assign requested address when agent tries to start

[#1]

Hello. I am standing up a 6 node cluster, 6 datanodes, one of which servers as our name node.

All firewalls have been disabled.

All host files are configured correctly (the entries missing the TLD were added based on feedback about the same issue on this forum)

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 localdomain
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.251.220.41   bcs-ula-hadoop1.ula.mydomain.net bcs-ula-hadoop1 bcs-ula-hadoop1.ula.mydomain
10.251.220.42   bcs-ula-hadoop2.ula.mydomain.net bcs-ula-hadoop2 bcs-ula-hadoop2.ula.mydomain
10.251.220.43   bcs-ula-hadoop3.ula.mydomain.net bcs-ula-hadoop3 bcs-ula-hadoop3.ula.mydomain
10.251.220.44   bcs-ula-hadoop4.ula.mydomain.net bcs-ula-hadoop4 bcs-ula-hadoop4.ula.mydomain
10.251.220.45   bcs-ula-hadoop5.ula.mydomain.net bcs-ula-hadoop5 bcs-ula-hadoop5.ula.mydomain
10.251.220.46   bcs-ula-hadoop6.ula.mydomain.net bcs-ula-hadoop6 bcs-ula-hadoop6.ula.mydomain

NSSwitch.conf has been verified to check files before checking DNS.

All hosts have DNS configured properly, A and PTR records are configured correctly, and queries respond with expected results. All hosts can communicate with one another.

I have verified that nothing is using port 9000 or 9001. Neither port is blocked by a firewall at either the host (as shown previously) or within the network.

I installed CDH manager on hadoop1, and it installed with no problem. I launched the web based installer, and this is when things started to go weird. I allowed root ssh access on every node. I also ensured that they all use the same password. Again, this was manually verified and is known to be working.

I selected installation using parcels, and using root login with password.

The base isntallation appears to succeed. However once the agent tries to start up, we get the following error:

/tmp/scm_prepare_node.m1QZtmaj 
using SSH_CLIENT to get the SCM hostname: 10.251.220.41 54009 22 
opening logging file descriptor 
Starting installation script...
Acquiring installation lock...
BEGIN flock 4 
END (0) 
Detecting root privileges...
effective UID is 0 
Detecting distribution...
BEGIN grep Tikanga /etc/redhat-release 
END (1) 
BEGIN grep 'CentOS release 5' /etc/redhat-release 
END (1) 
BEGIN grep 'Scientific Linux release 5' /etc/redhat-release 
END (1) 
BEGIN grep Santiago /etc/redhat-release 
END (1) 
BEGIN grep 'CentOS Linux release 6' /etc/redhat-release 
END (1) 
BEGIN grep 'CentOS release 6' /etc/redhat-release 
END (0) 
CentOS release 6.6 (Final) 
/etc/redhat-release ==> CentOS 6 
Detecting Cloudera Manager Server...
BEGIN host -t PTR 10.251.220.41 
41.220.251.10.in-addr.arpa domain name pointer bcs-ula-hadoop1.ula.mydomain.net. 
END (0) 
using bcs-ula-hadoop1.ula.mydomain.net as scm server hostname 
BEGIN which python 
END (0) 
/usr/bin/python 
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' bcs-ula-hadoop1.ula.mydomain.net 7182 
END (0) 
BEGIN which wget 
/usr/bin/wget 
END (0) 
BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo 
bcs-ula-hadoop1.novalocal 
END (0) 
Installing package repositories...
validating format of repository file /tmp/scm_prepare_node.m1QZtmaj/repos/rhel6/cloudera-manager.repo 
installing repository file /tmp/scm_prepare_node.m1QZtmaj/repos/rhel6/cloudera-manager.repo 
repository file /tmp/scm_prepare_node.m1QZtmaj/repos/rhel6/cloudera-manager.repo installed 
Refreshing package metadata...
BEGIN yum clean all 
Loaded plugins: fastestmirror 
Cleaning repos: atlas-repo-100-epel_6_x86_64 
: atlas-repo-12967-seo-centos6.4-x86_64-server-12-5-2013 
: atlas-repo-12968-seo-centos6.5-x86_64-server-10-15-2014 
: atlas-repo-5758-pas-centos6.4-x86_64-server-2013-03-09 
: atlas-repo-5761-pas-centos6.5-x86_64-server-2013-12-01 
: atlas-repo-96-centos_6_x86_64 cloudera-manager 
Cleaning up Everything 
END (0) 
BEGIN rm -Rf /var/cache/yum/CentOS 6.x update /var/cache/yum/Cloud Init /var/cache/yum/EPEL 6.x /var/cache/yum/x86_64 
END (0) 
BEGIN yum makecache 
Loaded plugins: fastestmirror 
Metadata Cache Created 
END (0) 
Installing jdk package...
BEGIN yum list installed jdk 
Loaded plugins: fastestmirror 
Installed Packages 
jdk.x86_64 2000:1.6.0_31-fcs @cloudera-manager 
END (0) 
BEGIN echo jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep jdk 
jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons 
END (0) 
BEGIN yum info jdk 
Loaded plugins: fastestmirror 
Determining fastest mirrors 
Installed Packages 
Name : jdk 
Arch : x86_64 
Epoch : 2000 
Version : 1.6.0_31 
Release : fcs 
Size : 143 M 
Repo : installed 
From repo : cloudera-manager 
Summary : Java(TM) Platform Standard Edition Development Kit 
URL : http://java.sun.com/ 
License : Copyright (c) 2011, Oracle and/or its affiliates. All rights 
: reserved. Also under other license(s) as shown at the Description 
: field. 
Description : The Java Platform Standard Edition Development Kit (JDK) includes 
: both the runtime environment (Java virtual machine, the Java 
: platform classes and supporting files) and development tools 
: (compilers, debuggers, tool libraries and other tools). 
: 
: The JDK is a development environment for building applications, 
: applets and components that can be deployed with the Java Platform 
: Standard Edition Runtime Environment. 

END (0) 
BEGIN yum -y install jdk.x86_64 
Loaded plugins: fastestmirror 
Setting up Install Process 
Loading mirror speeds from cached hostfile 
Package 2000:jdk-1.6.0_31-fcs.x86_64 already installed and latest version 
Nothing to do 
END (0) 
remote package jdk installed 
Installing oracle-j2sdk1.7 package...
BEGIN yum list installed oracle-j2sdk1.7 
Loaded plugins: fastestmirror 
Installed Packages 
oracle-j2sdk1.7.x86_64 1.7.0+update67-1 @cloudera-manager 
END (0) 
BEGIN echo jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep oracle-j2sdk1.7 
jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons 
END (0) 
BEGIN yum info oracle-j2sdk1.7 
Loaded plugins: fastestmirror 
Loading mirror speeds from cached hostfile 
Installed Packages 
Name : oracle-j2sdk1.7 
Arch : x86_64 
Version : 1.7.0+update67 
Release : 1 
Size : 279 M 
Repo : installed 
From repo : cloudera-manager 
Summary : no description given 
URL : http://example.com/no-uri-given 
License : unknown 
Description : no description given 

END (0) 
BEGIN yum -y install oracle-j2sdk1.7.x86_64 
Loaded plugins: fastestmirror 
Setting up Install Process 
Loading mirror speeds from cached hostfile 
Package oracle-j2sdk1.7-1.7.0+update67-1.x86_64 already installed and latest version 
Nothing to do 
END (0) 
remote package oracle-j2sdk1.7 installed 
Installing cloudera-manager-agent package...
BEGIN yum list installed cloudera-manager-agent 
Loaded plugins: fastestmirror 
Error: No matching Packages to list 
END (1) 
BEGIN yum info cloudera-manager-agent 
Loaded plugins: fastestmirror 
Loading mirror speeds from cached hostfile 
Available Packages 
Name : cloudera-manager-agent 
Arch : x86_64 
Version : 5.4.1 
Release : 1.cm541.p0.197.el6 
Size : 4.6 M 
Repo : cloudera-manager 
Summary : The Cloudera Manager Agent 
URL : http://www.cloudera.com 
License : Proprietary 
Description : The Cloudera Manager Agent. 
: 
: The Agent is deployed to machines running services managed by 
: Cloudera Manager. 

END (0) 
Version : 5.4.1 
Release : 1.cm541.p0.197.el6 
BEGIN yum -y install cloudera-manager-agent 
Loaded plugins: fastestmirror 
Setting up Install Process 
Loading mirror speeds from cached hostfile 
Resolving Dependencies 
--> Running transaction check 
---> Package cloudera-manager-agent.x86_64 0:5.4.1-1.cm541.p0.197.el6 will be installed 
--> Finished Dependency Resolution 

Dependencies Resolved 

================================================================================ 
Package Arch Version Repository Size 
================================================================================ 
Installing: 
cloudera-manager-agent x86_64 5.4.1-1.cm541.p0.197.el6 cloudera-manager 4.6 M 

Transaction Summary 
================================================================================ 
Install 1 Package(s) 

Total download size: 4.6 M 
Installed size: 32 M 
Downloading Packages: 
Running rpm_check_debug 
Running Transaction Test 
Transaction Test Succeeded 
Running Transaction 
Installing : cloudera-manager-agent-5.4.1-1.cm541.p0.197.el6.x86_64 1/1 
Verifying : cloudera-manager-agent-5.4.1-1.cm541.p0.197.el6.x86_64 1/1 

Installed: 
cloudera-manager-agent.x86_64 0:5.4.1-1.cm541.p0.197.el6 

Complete! 
END (0) 
remote package cloudera-manager-agent installed 
Installing cloudera-manager-daemons package...
BEGIN yum list installed cloudera-manager-daemons 
Loaded plugins: fastestmirror 
Installed Packages 
cloudera-manager-daemons.x86_64 5.4.1-1.cm541.p0.197.el6 @cloudera-manager 
END (0) 
BEGIN echo jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons 
jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons 
END (0) 
BEGIN yum info cloudera-manager-daemons 
Loaded plugins: fastestmirror 
Loading mirror speeds from cached hostfile 
Installed Packages 
Name : cloudera-manager-daemons 
Arch : x86_64 
Version : 5.4.1 
Release : 1.cm541.p0.197.el6 
Size : 902 M 
Repo : installed 
From repo : cloudera-manager 
Summary : Provides daemons for monitoring Hadoop and related tools. 
URL : http://www.cloudera.com 
License : Proprietary 
Description : This package includes daemons for monitoring and managing Hadoop. 

END (0) 
Version : 5.4.1 
Release : 1.cm541.p0.197.el6 
BEGIN yum -y install cloudera-manager-daemons 
Loaded plugins: fastestmirror 
Setting up Install Process 
Loading mirror speeds from cached hostfile 
Package cloudera-manager-daemons-5.4.1-1.cm541.p0.197.el6.x86_64 already installed and latest version 
Nothing to do 
END (0) 
remote package cloudera-manager-daemons installed 
Installing Unlimited Strength Encryption policy files.
BEGIN rpm -ql jdk | grep "/usr/java/jdk1.6" | sort | head -n 1 
/usr/java/jdk1.6.0_31 
END (0) 
BEGIN rpm -ql oracle-j2sdk1.7 | grep "/usr/java/jdk1.7" | sort | head -n 1 
/usr/java/jdk1.7.0_67-cloudera 
END (0) 
Java 6 prefix is /usr/java/jdk1.6.0_31 
Java 7 prefix is /usr/java/jdk1.7.0_67-cloudera 
Installing unlimited strength US_export_policy.jar for Java 6 
BEGIN cp /tmp/scm_prepare_node.m1QZtmaj/US_export_policy.jar.6 /usr/java/jdk1.6.0_31/jre/lib/security/US_export_policy.jar 
END (0) 
Installing unlimited strength local_policy.jar for Java 6 
BEGIN cp /tmp/scm_prepare_node.m1QZtmaj/local_policy.jar.6 /usr/java/jdk1.6.0_31/jre/lib/security/local_policy.jar 
END (0) 
Installing unlimited strength US_export_policy.jar for Java 7 
BEGIN cp /tmp/scm_prepare_node.m1QZtmaj/US_export_policy.jar.7 /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/US_export_policy.jar 
END (0) 
Installing unlimited strength local_policy.jar for Java 7 
BEGIN cp /tmp/scm_prepare_node.m1QZtmaj/local_policy.jar.7 /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/local_policy.jar 
END (0) 
Configuring Cloudera Manager Agent...
BEGIN grep server_host=bcs-ula-hadoop1.ula.mydomain.net /etc/cloudera-scm-agent/config.ini 
END (1) 
BEGIN sed -e 's/\(server_host=\).*/\1bcs-ula-hadoop1.ula.mydomain.net/' -i /etc/cloudera-scm-agent/config.ini 
END (0) 
scm agent configured 
Starting Cloudera Manager Agent...
BEGIN /sbin/service cloudera-scm-agent status | grep running 
END (1) 
BEGIN /sbin/service cloudera-scm-agent start 
Starting cloudera-scm-agent: [FAILED] 
END (1) 
agent logs: 
BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/' 
>>/usr/lib64/cmf/agent/src/cmf/parcel.py:17: DeprecationWarning: the sets module is deprecated 
>> from sets import Set 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO SCM Agent Version: 5.4.1 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Adding env vars that start with CMF_AGENT_ 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log 
>>/usr/lib64/cmf/agent/src/cmf/parcel.py:17: DeprecationWarning: the sets module is deprecated 
>> from sets import Set 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO SCM Agent Version: 5.4.1 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Adding env vars that start with CMF_AGENT_ 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log 
END (0) 
BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/' 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: cpu 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: cpuacct 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: memory 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: blkio 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/memory 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpu 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/blkio 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Setting up supervisord event monitor. 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using timeout of 2.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using join timeout of 0.100000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using tolerance of 60.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4'] 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread dns_names INFO Using timeout of 2.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread ntp_monitor INFO Using timeout of 2.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread __init__ INFO Importing metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Supervisor version: 3.0 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Connecting to previous supervisor: agent-23870-1432904810. 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes. 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes. 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STARTING 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread _cplogging INFO [29/May/2015:13:48:46] ENGINE Started monitor thread '_TimeoutMonitor'. 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging ERROR [29/May/2015:13:48:46] ENGINE Error in HTTP server: shutting down 
>>Traceback (most recent call last): 
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread 
>> self.httpserver.start() 
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start 
>> raise socket.error(msg) 
>>error: No socket could be created on ('bcs-ula-hadoop1.ula.mydomain.net', 9000) -- [Errno 99] Cannot assign requested address 
>> 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STOPPING 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('bcs-ula-hadoop1.ula.mydomain.net', 9000)) already shut down 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Stopped thread '_TimeoutMonitor'. 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STOPPED 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus EXITING 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus EXITED 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: cpu 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: cpuacct 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: memory 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: blkio 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/memory 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpu 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/blkio 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Setting up supervisord event monitor. 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using timeout of 2.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using join timeout of 0.100000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using tolerance of 60.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4'] 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread dns_names INFO Using timeout of 2.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread ntp_monitor INFO Using timeout of 2.000000 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 
>>[29/May/2015 13:48:45 +0000] 29108 MainThread __init__ INFO Importing metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Supervisor version: 3.0 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Connecting to previous supervisor: agent-23870-1432904810. 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes. 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes. 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STARTING 
>>[29/May/2015 13:48:46 +0000] 29108 MainThread _cplogging INFO [29/May/2015:13:48:46] ENGINE Started monitor thread '_TimeoutMonitor'. 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging ERROR [29/May/2015:13:48:46] ENGINE Error in HTTP server: shutting down 
>>Traceback (most recent call last): 
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread 
>> self.httpserver.start() 
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start 
>> raise socket.error(msg) 
>>error: No socket could be created on ('bcs-ula-hadoop1.ula.mydomain.net', 9000) -- [Errno 99] Cannot assign requested address 
>> 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STOPPING 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('bcs-ula-hadoop1.ula.mydomain.net', 9000)) already shut down 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Stopped thread '_TimeoutMonitor'. 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STOPPED 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus EXITING 
>>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus EXITED 
END (0) 
end of agent logs. 
scm agent could not be started, giving up 
waiting for rollback request 

With that info, can anyone provide some help? As of right now I can't find a single prereq that isn't met. I've done every possible fix that I could find searching for the error with no luck. Does anyone have any ideas?

I have verified that port 9000 CAN BE opened by using nc, and it can be connected to by other hosts. This suggests the issue lies with CDH and not with our host's config. Nothing is blocking the port from being accessed or opened.

On manager/namenode:

[root@bcs-ula-hadoop6 ~]# nc -l 9000

On another host:

[root@bcs-ula-hadoop6 ~]# telnet bcs-ula-hadoop1 9000
Trying 10.251.220.41...
Connected to bcs-ula-hadoop1.
Escape character is '^]'.
^]

telnet> q
Connection closed.
[root@bcs-ula-hadoop6 ~]#

[#2]

Could you tell us the output of the following script from one of the nodes?
$ python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"

Also, as of C5.4, IPv6 is not supported, so could you disable IPv6 permanently

# echo net.ipv6.conf.all.disable_ipv6=1 >> /etc/sysctl.conf
# sysctl -w net.ipv6.conf.all.disable_ipv6=1

Then try again and let us know if it helped.

[#3]

I have a similar problem,log is here:

>>[24/Aug/2016 21:54:58 +0000] 13517 MainThread _cplogging INFO [24/Aug/2016:21:54:58] ENGINE Started monitor thread '_TimeoutMonitor'.
>>[24/Aug/2016 21:54:58 +0000] 13517 HTTPServer Thread-2 _cplogging ERROR [24/Aug/2016:21:54:58] ENGINE Error in HTTP server: shutting down
>>Traceback (most recent call last):
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/CherryPy-3.2.2-py2.7.egg/cherrypy/process/servers.py", line 187, in _start_http_thread
>> self.httpserver.start()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/CherryPy-3.2.2-py2.7.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start
>> raise socket.error(msg)
>>error: No socket could be created on ('change.example.com', 9000) -- [Errno 99] Cannot assign requested address
>>

[root@change ~]# python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"
change.example.com
202.102.110.203

but my hosts config is this:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.111 hadoop1.example.com hadoop1
192.168.0.112 hadoop2.example.com hadoop2
192.168.0.113 hadoop3.example.com hadoop3
192.168.0.114 hadoop4.example.com hadoop4
192.168.0.115 hadoop5.example.com hadoop5
192.168.0.110 base.example.com base

any idea?

[#4]

resolve this by set hostname,centos7 hostname is not set by /etc/sysconfig/network.

[#5]

the host change.example.com ,I have ever config,but now it's should not exists...