Failed to start Cloudera Manager Agent on AWS with error: No socket could be created on 9000 port

[#1]

Hi Guys,

I am stuck with CDH5 installation on an AWS provisioned Red Hat node - single machine installation (no cluster)

Briefly, I followed following pre-installation steps:

1. Updated /etc/hosts

2. Disabled selinux

3. Disabled iptables

4. Restarted nodes

I am installing CDH using ec2-user

I am getting following error:

[18/Aug/2015 03:35:04 +0000] 25479 HTTPServer Thread-2 _cplogging ERROR [18/Aug/2015:03:35:04] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread
self.httpserver.start()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start
raise socket.error(msg)
error: No socket could be created on ('ec2-52-18-238-34.eu-west-1.compute.amazonaws.com', 9000) -- [Errno 99] Cannot assign requested address

Please find full log below:

--------------------------------

Starting Cloudera Manager Agent...

BEGIN /sbin/service cloudera-scm-agent status | grep running 
END (1) 
BEGIN sudo /sbin/service cloudera-scm-agent start 
Starting cloudera-scm-agent: [60G[[0;31mFAILED[0;39m] 
END (1) 
agent logs: 
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/' 
>>/usr/lib64/cmf/agent/src/cmf/parcel.py:17: DeprecationWarning: the sets module is deprecated 
>> from sets import Set 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO SCM Agent Version: 5.4.3 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Adding env vars that start with CMF_AGENT_ 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log 
>>/usr/lib64/cmf/agent/src/cmf/parcel.py:17: DeprecationWarning: the sets module is deprecated 
>> from sets import Set 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO SCM Agent Version: 5.4.3 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Adding env vars that start with CMF_AGENT_ 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log 
END (0) 
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/' 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Agent is running on a public cloud. Public hostname: ec2-52-18-238-34.eu-west-1.compute.amazonaws.com 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Found cgroups subsystem: cpu 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Found cgroups subsystem: cpuacct 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Found cgroups subsystem: memory 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Found cgroups subsystem: blkio 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/memory 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpu 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/blkio 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Setting up supervisord event monitor. 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Using timeout of 2.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Using join timeout of 0.100000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Using tolerance of 60.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4'] 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread dns_names INFO Using timeout of 2.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread ntp_monitor INFO Using timeout of 2.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread __init__ INFO Importing metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Supervisor version: 3.0 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Connecting to previous supervisor: agent-25479-1439883302. 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes. 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes. 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus STARTING 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Started monitor thread '_TimeoutMonitor'. 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging ERROR [18/Aug/2015:03:37:29] ENGINE Error in HTTP server: shutting down 
>>Traceback (most recent call last): 
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread 
>> self.httpserver.start() 
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start 
>> raise socket.error(msg) 
>>error: No socket could be created on ('ec2-52-18-238-34.eu-west-1.compute.amazonaws.com', 9000) -- [Errno 99] Cannot assign requested address 
>> 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus STOPPING 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('ec2-52-18-238-34.eu-west-1.compute.amazonaws.com', 9000)) already shut down 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Stopped thread '_TimeoutMonitor'. 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus STOPPED 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus EXITING 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServe>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Agent is running on a public cloud. Public hostname: ec2-52-18-238-34.eu-west-1.compute.amazonaws.com 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Found cgroups subsystem: cpu 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Found cgroups subsystem: cpuacct 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Found cgroups subsystem: memory 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Found cgroups subsystem: blkio 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/memory 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpu 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/blkio 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Setting up supervisord event monitor. 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Using timeout of 2.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Using join timeout of 0.100000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Using tolerance of 60.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4'] 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread dns_names INFO Using timeout of 2.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread ntp_monitor INFO Using timeout of 2.000000 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 
>>[18/Aug/2015 03:37:28 +0000] 27040 MainThread __init__ INFO Importing metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Supervisor version: 3.0 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread agent INFO Connecting to previous supervisor: agent-25479-1439883302. 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes. 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes. 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus STARTING 
>>[18/Aug/2015 03:37:29 +0000] 27040 MainThread _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Started monitor thread '_TimeoutMonitor'. 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging ERROR [18/Aug/2015:03:37:29] ENGINE Error in HTTP server: shutting down 
>>Traceback (most recent call last): 
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread 
>> self.httpserver.start() 
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start 
>> raise socket.error(msg) 
>>error: No socket could be created on ('ec2-52-18-238-34.eu-west-1.compute.amazonaws.com', 9000) -- [Errno 99] Cannot assign requested address 
>> 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus STOPPING 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('ec2-52-18-238-34.eu-west-1.compute.amazonaws.com', 9000)) already shut down 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Stopped thread '_TimeoutMonitor'. 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus STOPPED 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus EXITING 
>>[18/Aug/2015 03:37:29 +0000] 27040 HTTPServer Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus EXITED 
r Thread-2 _cplogging INFO [18/Aug/2015:03:37:29] ENGINE Bus EXITED 
END (0) 
end of agent logs. 
scm agent could not be started, giving up 
waiting for rollback request 

----------------------------------------------------------------------

Any leads on this will be highly appriciated.

[#2]

Did some more research on the errors on python scripts. Following are the error lines from the scripts:

Python files error lines:

1. vi /usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py - line 187

def _start_http_thread(self):

        """HTTP servers MUST be running in new threads, so that the

        main thread persists to receive KeyboardInterrupt's. If an

        exception is raised in the httpserver's thread then it's

        trapped here, and the bus (and therefore our httpserver)

        are shut down.

        """

        try:

            self.httpserver.start()

        except KeyboardInterrupt:

            self.bus.log("<Ctrl-C> hit: shutting down HTTP server")

            self.interrupt = sys.exc_info()[1]

            self.bus.exit()

        except SystemExit:

            self.bus.log("SystemExit raised: shutting down HTTP server")

            self.interrupt = sys.exc_info()[1]

            self.bus.exit()

            raise

        except:

            self.interrupt = sys.exc_info()[1]

            self.bus.log("Error in HTTP server: shutting down",

                         traceback=True, level=40)

            self.bus.exit()

            raise

2.   vi /usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py – line 1825

try:

                info = socket.getaddrinfo(host, port, socket.AF_UNSPEC,

                                          socket.SOCK_STREAM, 0, socket.AI_PASSIVE)

            except socket.gaierror:

                if ':' in self.bind_addr[0]:

                    info = [(socket.AF_INET6, socket.SOCK_STREAM,

                             0, "", self.bind_addr + (0, 0))]

                else:

                    info = [(socket.AF_INET, socket.SOCK_STREAM,

                             0, "", self.bind_addr)]

        self.socket = None

        msg = "No socket could be created"

        for res in info:

            af, socktype, proto, canonname, sa = res

            try:

                self.bind(af, socktype, proto)

            except socket.error, ex:

                msg = "%s on %s -- %s" % (msg, self.bind_addr, ex)

                if self.socket:

                    self.socket.close()

                self.socket = None

                continue

            break

        if not self.socket:

            raise socket.error(msg)

[#3]

Try running "netstat -anp | grep 9000" and see if that tells you who holds the port 9000. Once you figure out how to stop that process, you can try starting the agent again.

[#4]

Hi gautam,

Thanks for your reply.. I already tried that.. nothing is holding up that port.

I am using RHEL-6.6_HVM_GA-20150128-x86_64-1-Hourly2-GP2 from Community AMI to install CDH

[#5]

Can you run a simple python command to see if it can bind to 9000? This will eliminate CDH/CM from the equation.

# python -m SimpleHTTPServer 9000

If this works, then the port is available and issue is with the agent.

If it fails, then the kernel has the port open for a process which netsat doesn't know about. The only solution would be to reboot the host, then try again.

[#6]

Hi Gautam,

script python -m SimpleHTTPServer 9000 got stuck.. did not return any result. I restarted the machine but same result.
I tried installing this on Ubuntu 14.04 LTS and it got installed with no errors. Looks like some issue with the AMI.

[#7]

 My bad.. it worked coz I was trying on single instance.. Back to RHEL

[#8 · Accepted solution]

got the issue..

I was modifying /etc/hosts file with following information:

Public_IP Public_DNS Private_DNS Private_IP Alias1

Public_IP Public_DNS Private_DNS Private_IP Alias2

Public_IP Public_DNS Private_DNS Private_IP Alias3

I just commented out the stuff and my /etc/hosts file is virgin:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

And this fixed the issue.. strange.. if someone could explain it'll be great.

[#9]

I am running into the same issue. Can you help me on  how to solving this isseu?

[#10]

What are the debugging steps you have performed on the error? Have you checked /etc/hosts file?