Hello. I am standing up a 6 node cluster, 6 datanodes, one of which servers as our name node.
All firewalls have been disabled.
All host files are configured correctly (the entries missing the TLD were added based on feedback about the same issue on this forum)
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 localdomain ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.251.220.41 bcs-ula-hadoop1.ula.mydomain.net bcs-ula-hadoop1 bcs-ula-hadoop1.ula.mydomain 10.251.220.42 bcs-ula-hadoop2.ula.mydomain.net bcs-ula-hadoop2 bcs-ula-hadoop2.ula.mydomain 10.251.220.43 bcs-ula-hadoop3.ula.mydomain.net bcs-ula-hadoop3 bcs-ula-hadoop3.ula.mydomain 10.251.220.44 bcs-ula-hadoop4.ula.mydomain.net bcs-ula-hadoop4 bcs-ula-hadoop4.ula.mydomain 10.251.220.45 bcs-ula-hadoop5.ula.mydomain.net bcs-ula-hadoop5 bcs-ula-hadoop5.ula.mydomain 10.251.220.46 bcs-ula-hadoop6.ula.mydomain.net bcs-ula-hadoop6 bcs-ula-hadoop6.ula.mydomain
NSSwitch.conf has been verified to check files before checking DNS.
All hosts have DNS configured properly, A and PTR records are configured correctly, and queries respond with expected results. All hosts can communicate with one another.
I have verified that nothing is using port 9000 or 9001. Neither port is blocked by a firewall at either the host (as shown previously) or within the network.
I installed CDH manager on hadoop1, and it installed with no problem. I launched the web based installer, and this is when things started to go weird. I allowed root ssh access on every node. I also ensured that they all use the same password. Again, this was manually verified and is known to be working.
I selected installation using parcels, and using root login with password.
The base isntallation appears to succeed. However once the agent tries to start up, we get the following error:
/tmp/scm_prepare_node.m1QZtmaj using SSH_CLIENT to get the SCM hostname: 10.251.220.41 54009 22 opening logging file descriptor Starting installation script... Acquiring installation lock... BEGIN flock 4 END (0) Detecting root privileges... effective UID is 0 Detecting distribution... BEGIN grep Tikanga /etc/redhat-release END (1) BEGIN grep 'CentOS release 5' /etc/redhat-release END (1) BEGIN grep 'Scientific Linux release 5' /etc/redhat-release END (1) BEGIN grep Santiago /etc/redhat-release END (1) BEGIN grep 'CentOS Linux release 6' /etc/redhat-release END (1) BEGIN grep 'CentOS release 6' /etc/redhat-release END (0) CentOS release 6.6 (Final) /etc/redhat-release ==> CentOS 6 Detecting Cloudera Manager Server... BEGIN host -t PTR 10.251.220.41 41.220.251.10.in-addr.arpa domain name pointer bcs-ula-hadoop1.ula.mydomain.net. END (0) using bcs-ula-hadoop1.ula.mydomain.net as scm server hostname BEGIN which python END (0) /usr/bin/python BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' bcs-ula-hadoop1.ula.mydomain.net 7182 END (0) BEGIN which wget /usr/bin/wget END (0) BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo bcs-ula-hadoop1.novalocal END (0) Installing package repositories... validating format of repository file /tmp/scm_prepare_node.m1QZtmaj/repos/rhel6/cloudera-manager.repo installing repository file /tmp/scm_prepare_node.m1QZtmaj/repos/rhel6/cloudera-manager.repo repository file /tmp/scm_prepare_node.m1QZtmaj/repos/rhel6/cloudera-manager.repo installed Refreshing package metadata... BEGIN yum clean all Loaded plugins: fastestmirror Cleaning repos: atlas-repo-100-epel_6_x86_64 : atlas-repo-12967-seo-centos6.4-x86_64-server-12-5-2013 : atlas-repo-12968-seo-centos6.5-x86_64-server-10-15-2014 : atlas-repo-5758-pas-centos6.4-x86_64-server-2013-03-09 : atlas-repo-5761-pas-centos6.5-x86_64-server-2013-12-01 : atlas-repo-96-centos_6_x86_64 cloudera-manager Cleaning up Everything END (0) BEGIN rm -Rf /var/cache/yum/CentOS 6.x update /var/cache/yum/Cloud Init /var/cache/yum/EPEL 6.x /var/cache/yum/x86_64 END (0) BEGIN yum makecache Loaded plugins: fastestmirror Metadata Cache Created END (0) Installing jdk package... BEGIN yum list installed jdk Loaded plugins: fastestmirror Installed Packages jdk.x86_64 2000:1.6.0_31-fcs @cloudera-manager END (0) BEGIN echo jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep jdk jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons END (0) BEGIN yum info jdk Loaded plugins: fastestmirror Determining fastest mirrors Installed Packages Name : jdk Arch : x86_64 Epoch : 2000 Version : 1.6.0_31 Release : fcs Size : 143 M Repo : installed From repo : cloudera-manager Summary : Java(TM) Platform Standard Edition Development Kit URL : http://java.sun.com/ License : Copyright (c) 2011, Oracle and/or its affiliates. All rights : reserved. Also under other license(s) as shown at the Description : field. Description : The Java Platform Standard Edition Development Kit (JDK) includes : both the runtime environment (Java virtual machine, the Java : platform classes and supporting files) and development tools : (compilers, debuggers, tool libraries and other tools). : : The JDK is a development environment for building applications, : applets and components that can be deployed with the Java Platform : Standard Edition Runtime Environment. END (0) BEGIN yum -y install jdk.x86_64 Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile Package 2000:jdk-1.6.0_31-fcs.x86_64 already installed and latest version Nothing to do END (0) remote package jdk installed Installing oracle-j2sdk1.7 package... BEGIN yum list installed oracle-j2sdk1.7 Loaded plugins: fastestmirror Installed Packages oracle-j2sdk1.7.x86_64 1.7.0+update67-1 @cloudera-manager END (0) BEGIN echo jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep oracle-j2sdk1.7 jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons END (0) BEGIN yum info oracle-j2sdk1.7 Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Installed Packages Name : oracle-j2sdk1.7 Arch : x86_64 Version : 1.7.0+update67 Release : 1 Size : 279 M Repo : installed From repo : cloudera-manager Summary : no description given URL : http://example.com/no-uri-given License : unknown Description : no description given END (0) BEGIN yum -y install oracle-j2sdk1.7.x86_64 Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile Package oracle-j2sdk1.7-1.7.0+update67-1.x86_64 already installed and latest version Nothing to do END (0) remote package oracle-j2sdk1.7 installed Installing cloudera-manager-agent package... BEGIN yum list installed cloudera-manager-agent Loaded plugins: fastestmirror Error: No matching Packages to list END (1) BEGIN yum info cloudera-manager-agent Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Available Packages Name : cloudera-manager-agent Arch : x86_64 Version : 5.4.1 Release : 1.cm541.p0.197.el6 Size : 4.6 M Repo : cloudera-manager Summary : The Cloudera Manager Agent URL : http://www.cloudera.com License : Proprietary Description : The Cloudera Manager Agent. : : The Agent is deployed to machines running services managed by : Cloudera Manager. END (0) Version : 5.4.1 Release : 1.cm541.p0.197.el6 BEGIN yum -y install cloudera-manager-agent Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package cloudera-manager-agent.x86_64 0:5.4.1-1.cm541.p0.197.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: cloudera-manager-agent x86_64 5.4.1-1.cm541.p0.197.el6 cloudera-manager 4.6 M Transaction Summary ================================================================================ Install 1 Package(s) Total download size: 4.6 M Installed size: 32 M Downloading Packages: Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : cloudera-manager-agent-5.4.1-1.cm541.p0.197.el6.x86_64 1/1 Verifying : cloudera-manager-agent-5.4.1-1.cm541.p0.197.el6.x86_64 1/1 Installed: cloudera-manager-agent.x86_64 0:5.4.1-1.cm541.p0.197.el6 Complete! END (0) remote package cloudera-manager-agent installed Installing cloudera-manager-daemons package... BEGIN yum list installed cloudera-manager-daemons Loaded plugins: fastestmirror Installed Packages cloudera-manager-daemons.x86_64 5.4.1-1.cm541.p0.197.el6 @cloudera-manager END (0) BEGIN echo jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons jdk oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons END (0) BEGIN yum info cloudera-manager-daemons Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Installed Packages Name : cloudera-manager-daemons Arch : x86_64 Version : 5.4.1 Release : 1.cm541.p0.197.el6 Size : 902 M Repo : installed From repo : cloudera-manager Summary : Provides daemons for monitoring Hadoop and related tools. URL : http://www.cloudera.com License : Proprietary Description : This package includes daemons for monitoring and managing Hadoop. END (0) Version : 5.4.1 Release : 1.cm541.p0.197.el6 BEGIN yum -y install cloudera-manager-daemons Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile Package cloudera-manager-daemons-5.4.1-1.cm541.p0.197.el6.x86_64 already installed and latest version Nothing to do END (0) remote package cloudera-manager-daemons installed Installing Unlimited Strength Encryption policy files. BEGIN rpm -ql jdk | grep "/usr/java/jdk1.6" | sort | head -n 1 /usr/java/jdk1.6.0_31 END (0) BEGIN rpm -ql oracle-j2sdk1.7 | grep "/usr/java/jdk1.7" | sort | head -n 1 /usr/java/jdk1.7.0_67-cloudera END (0) Java 6 prefix is /usr/java/jdk1.6.0_31 Java 7 prefix is /usr/java/jdk1.7.0_67-cloudera Installing unlimited strength US_export_policy.jar for Java 6 BEGIN cp /tmp/scm_prepare_node.m1QZtmaj/US_export_policy.jar.6 /usr/java/jdk1.6.0_31/jre/lib/security/US_export_policy.jar END (0) Installing unlimited strength local_policy.jar for Java 6 BEGIN cp /tmp/scm_prepare_node.m1QZtmaj/local_policy.jar.6 /usr/java/jdk1.6.0_31/jre/lib/security/local_policy.jar END (0) Installing unlimited strength US_export_policy.jar for Java 7 BEGIN cp /tmp/scm_prepare_node.m1QZtmaj/US_export_policy.jar.7 /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/US_export_policy.jar END (0) Installing unlimited strength local_policy.jar for Java 7 BEGIN cp /tmp/scm_prepare_node.m1QZtmaj/local_policy.jar.7 /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/local_policy.jar END (0) Configuring Cloudera Manager Agent... BEGIN grep server_host=bcs-ula-hadoop1.ula.mydomain.net /etc/cloudera-scm-agent/config.ini END (1) BEGIN sed -e 's/\(server_host=\).*/\1bcs-ula-hadoop1.ula.mydomain.net/' -i /etc/cloudera-scm-agent/config.ini END (0) scm agent configured Starting Cloudera Manager Agent... BEGIN /sbin/service cloudera-scm-agent status | grep running END (1) BEGIN /sbin/service cloudera-scm-agent start Starting cloudera-scm-agent: [60G[[0;31mFAILED[0;39m] END (1) agent logs: BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/' >>/usr/lib64/cmf/agent/src/cmf/parcel.py:17: DeprecationWarning: the sets module is deprecated >> from sets import Set >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO SCM Agent Version: 5.4.1 >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Adding env vars that start with CMF_AGENT_ >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log >>/usr/lib64/cmf/agent/src/cmf/parcel.py:17: DeprecationWarning: the sets module is deprecated >> from sets import Set >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO SCM Agent Version: 5.4.1 >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Adding env vars that start with CMF_AGENT_ >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log END (0) BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/' >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: cpu >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: cpuacct >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: memory >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: blkio >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/memory >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpu >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/blkio >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Setting up supervisord event monitor. >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using timeout of 2.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using join timeout of 0.100000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using tolerance of 60.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4'] >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json >>[29/May/2015 13:48:45 +0000] 29108 MainThread dns_names INFO Using timeout of 2.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread ntp_monitor INFO Using timeout of 2.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 >>[29/May/2015 13:48:45 +0000] 29108 MainThread __init__ INFO Importing metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Supervisor version: 3.0 >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Connecting to previous supervisor: agent-23870-1432904810. >>[29/May/2015 13:48:46 +0000] 29108 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes. >>[29/May/2015 13:48:46 +0000] 29108 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes. >>[29/May/2015 13:48:46 +0000] 29108 MainThread _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STARTING >>[29/May/2015 13:48:46 +0000] 29108 MainThread _cplogging INFO [29/May/2015:13:48:46] ENGINE Started monitor thread '_TimeoutMonitor'. >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging ERROR [29/May/2015:13:48:46] ENGINE Error in HTTP server: shutting down >>Traceback (most recent call last): >> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread >> self.httpserver.start() >> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start >> raise socket.error(msg) >>error: No socket could be created on ('bcs-ula-hadoop1.ula.mydomain.net', 9000) -- [Errno 99] Cannot assign requested address >> >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STOPPING >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('bcs-ula-hadoop1.ula.mydomain.net', 9000)) already shut down >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Stopped thread '_TimeoutMonitor'. >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STOPPED >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus EXITING >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus EXITED >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: cpu >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: cpuacct >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: memory >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Found cgroups subsystem: blkio >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/memory >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpu >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct >>[29/May/2015 13:48:45 +0000] 29108 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/blkio >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Setting up supervisord event monitor. >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using timeout of 2.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using join timeout of 0.100000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Using tolerance of 60.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4'] >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json >>[29/May/2015 13:48:45 +0000] 29108 MainThread dns_names INFO Using timeout of 2.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread ntp_monitor INFO Using timeout of 2.000000 >>[29/May/2015 13:48:45 +0000] 29108 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 >>[29/May/2015 13:48:45 +0000] 29108 MainThread __init__ INFO Importing metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Supervisor version: 3.0 >>[29/May/2015 13:48:46 +0000] 29108 MainThread agent INFO Connecting to previous supervisor: agent-23870-1432904810. >>[29/May/2015 13:48:46 +0000] 29108 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes. >>[29/May/2015 13:48:46 +0000] 29108 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes. >>[29/May/2015 13:48:46 +0000] 29108 MainThread _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STARTING >>[29/May/2015 13:48:46 +0000] 29108 MainThread _cplogging INFO [29/May/2015:13:48:46] ENGINE Started monitor thread '_TimeoutMonitor'. >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging ERROR [29/May/2015:13:48:46] ENGINE Error in HTTP server: shutting down >>Traceback (most recent call last): >> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread >> self.httpserver.start() >> File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start >> raise socket.error(msg) >>error: No socket could be created on ('bcs-ula-hadoop1.ula.mydomain.net', 9000) -- [Errno 99] Cannot assign requested address >> >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STOPPING >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('bcs-ula-hadoop1.ula.mydomain.net', 9000)) already shut down >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Stopped thread '_TimeoutMonitor'. >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus STOPPED >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus EXITING >>[29/May/2015 13:48:46 +0000] 29108 HTTPServer Thread-2 _cplogging INFO [29/May/2015:13:48:46] ENGINE Bus EXITED END (0) end of agent logs. scm agent could not be started, giving up waiting for rollback request
With that info, can anyone provide some help? As of right now I can't find a single prereq that isn't met. I've done every possible fix that I could find searching for the error with no luck. Does anyone have any ideas?
I have verified that port 9000 CAN BE opened by using nc, and it can be connected to by other hosts. This suggests the issue lies with CDH and not with our host's config. Nothing is blocking the port from being accessed or opened.
On manager/namenode:
[root@bcs-ula-hadoop6 ~]# nc -l 9000
On another host:
[root@bcs-ula-hadoop6 ~]# telnet bcs-ula-hadoop1 9000 Trying 10.251.220.41... Connected to bcs-ula-hadoop1. Escape character is '^]'. ^] telnet> q Connection closed. [root@bcs-ula-hadoop6 ~]#
Created 06-03-2015 08:23 PM
Created 08-24-2016 07:36 AM
I have a similar problem,log is here:
>>[24/Aug/2016 21:54:58 +0000] 13517 MainThread _cplogging INFO [24/Aug/2016:21:54:58] ENGINE Started monitor thread '_TimeoutMonitor'.
>>[24/Aug/2016 21:54:58 +0000] 13517 HTTPServer Thread-2 _cplogging ERROR [24/Aug/2016:21:54:58] ENGINE Error in HTTP server: shutting down
>>Traceback (most recent call last):
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/CherryPy-3.2.2-py2.7.egg/cherrypy/process/servers.py", line 187, in _start_http_thread
>> self.httpserver.start()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/CherryPy-3.2.2-py2.7.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start
>> raise socket.error(msg)
>>error: No socket could be created on ('change.example.com', 9000) -- [Errno 99] Cannot assign requested address
>>
[root@change ~]# python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"
change.example.com
202.102.110.203
but my hosts config is this:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.111 hadoop1.example.com hadoop1
192.168.0.112 hadoop2.example.com hadoop2
192.168.0.113 hadoop3.example.com hadoop3
192.168.0.114 hadoop4.example.com hadoop4
192.168.0.115 hadoop5.example.com hadoop5
192.168.0.110 base.example.com base
any idea?
Created 08-24-2016 08:53 AM
Created 08-24-2016 07:38 AM