28785
DISCUSSIONS
102010
MEMBERS
3160
ARTICLES
Created 06-20-2016 05:11 AM
I have read through the community site and have tried to resolve the "waiting for heartbeat" error to no avail.
Here is what i have tried:
1. hostname
2. TLS set to 0
3. NTP installed
4. port 7182, 9000 and 9001 are running and can be reached
echo "quit" | nc -v (cm server) 7182
Here is the /var/log/cloudera-scm-agent/cloudera-scm-agent.log:
[20/Jun/2016 20:07:31 +0000] 6184 MainThread agent INFO Stopping agent...
[20/Jun/2016 20:07:31 +0000] 6184 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
[20/Jun/2016 20:07:31 +0000] 6184 MainThread agent INFO No processes are being managed; Supervisor will shutdown.
[20/Jun/2016 20:07:31 +0000] 6184 MainThread agent INFO Shutting down supervisord, pid 6219
[20/Jun/2016 20:07:32 +0000] 6184 MainThread agent INFO waiting for process to terminate...
[20/Jun/2016 20:07:32 +0000] 6184 MainThread agent INFO Successfully killed process with pid 6219
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus STOPPING
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('sg-master.dagupan.com', 9000)) shut down
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Stopped thread '_TimeoutMonitor'.
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus STOPPED
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus STOPPING
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('sg-master.dagupan.com', 9000)) already shut down
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE No thread running for None.
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus STOPPED
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus EXITING
[20/Jun/2016 20:07:32 +0000] 6184 MainThread _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus EXITED
[20/Jun/2016 20:07:32 +0000] 6184 MainThread agent INFO Cleaning up daemon
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent INFO Stopping agent...
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent INFO No extant cgroups; unmounting any cgroup roots
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent ERROR Shutdown callback failed.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/agent.py", line 2764, in stop
f()
File "/usr/lib64/python2.6/asyncore.py", line 394, in close
self.socket.close()
File "/usr/lib64/python2.6/asyncore.py", line 615, in close
os.close(self.fd)
OSError: [Errno 9] Bad file descriptor
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent INFO No processes are being managed; Supervisor will shutdown.
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent INFO Shutting down supervisord, pid 6219
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent ERROR Failed to kill process with pid 6219
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/agent.py", line 2819, in kill_process
os.kill(pid, signal.SIGTERM)
OSError: [Errno 3] No such process
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent ERROR Shutdown callback failed.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/agent.py", line 2764, in stop
f()
File "/usr/lib64/python2.6/asyncore.py", line 394, in close
self.socket.close()
File "/usr/lib64/python2.6/asyncore.py", line 615, in close
os.close(self.fd)
OSError: [Errno 9] Bad file descriptor
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus STOPPING
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('sg-master.dagupan.com', 9000)) already shut down
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE No thread running for None.
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus STOPPED
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus STOPPING
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('sg-master.dagupan.com', 9000)) already shut down
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE No thread running for None.
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus STOPPED
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus EXITING
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 _cplogging INFO [20/Jun/2016:20:07:32] ENGINE Bus EXITED
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent ERROR Shutdown callback failed.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/agent.py", line 2764, in stop
f()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/pyinotify-0.9.3-py2.6.egg/pyinotify.py", line 1424, in stop
self._pollobj.unregister(self._fd)
KeyError: 13
[20/Jun/2016 20:07:32 +0000] 6184 Dummy-14 agent INFO Cleaning up daemon
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO SCM Agent Version: 5.7.1
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Agent Protocol Version: 4
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Using Host ID: 26b964fa-f68c-4292-8cde-6ba66489016b
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Using directory: /var/run/cloudera-scm-agent
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Using supervisor binary path: /usr/lib64/cmf/agent/build/env/bin/supervisord
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Agent Logging Level: INFO
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO No command line vars
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're not using this database type)
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
[20/Jun/2016 20:07:35 +0000] 7616 MainThread agent INFO Agent starting as pid 7616 user root(0) group root(0).
[20/Jun/2016 20:07:37 +0000] 7616 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
[20/Jun/2016 20:07:37 +0000] 7616 MainThread cgroups INFO Found cgroups subsystem: cpu
[20/Jun/2016 20:07:37 +0000] 7616 MainThread cgroups INFO Found cgroups subsystem: cpuacct
[20/Jun/2016 20:07:37 +0000] 7616 MainThread cgroups INFO Found cgroups subsystem: memory
[20/Jun/2016 20:07:37 +0000] 7616 MainThread cgroups INFO Found cgroups subsystem: blkio
[20/Jun/2016 20:07:37 +0000] 7616 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/memory
[20/Jun/2016 20:07:37 +0000] 7616 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpu
[20/Jun/2016 20:07:37 +0000] 7616 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct
[20/Jun/2016 20:07:37 +0000] 7616 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/blkio
[20/Jun/2016 20:07:37 +0000] 7616 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True}
[20/Jun/2016 20:07:37 +0000] 7616 MainThread agent INFO Setting up supervisord event monitor.
[20/Jun/2016 20:07:37 +0000] 7616 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[20/Jun/2016 20:07:37 +0000] 7616 MainThread filesystem_map INFO Using timeout of 2.000000
[20/Jun/2016 20:07:37 +0000] 7616 MainThread filesystem_map INFO Using join timeout of 0.100000
[20/Jun/2016 20:07:37 +0000] 7616 MainThread filesystem_map INFO Using tolerance of 60.000000
[20/Jun/2016 20:07:37 +0000] 7616 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4']
[20/Jun/2016 20:07:37 +0000] 7616 MainThread kt_renewer INFO Agent wide credential cache set to /var/run/cloudera-scm-agent/krb5cc_cm_agent_0
[20/Jun/2016 20:07:37 +0000] 7616 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
[20/Jun/2016 20:07:37 +0000] 7616 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
[20/Jun/2016 20:07:37 +0000] 7616 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
[20/Jun/2016 20:07:37 +0000] 7616 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/tasktracker/schema.json
[20/Jun/2016 20:07:37 +0000] 7616 MainThread ntp_monitor INFO Using timeout of 2.000000
[20/Jun/2016 20:07:37 +0000] 7616 MainThread dns_names INFO Using timeout of 30.000000
[20/Jun/2016 20:07:37 +0000] 7616 MainThread __init__ INFO Created DNS monitor.
[20/Jun/2016 20:07:37 +0000] 7616 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
[20/Jun/2016 20:07:37 +0000] 7616 MainThread __init__ INFO Importing metric schema from file /usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/schema.json
[20/Jun/2016 20:07:39 +0000] 7616 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[20/Jun/2016 20:07:39 +0000] 7616 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[20/Jun/2016 20:07:39 +0000] 7616 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
[20/Jun/2016 20:07:39 +0000] 7616 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[20/Jun/2016 20:07:39 +0000] 7616 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
[20/Jun/2016 20:07:39 +0000] 7616 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[20/Jun/2016 20:07:39 +0000] 7616 MainThread agent ERROR Failed to connect to previous supervisor.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/agent.py", line 2037, in find_or_start_supervisor
self.get_supervisor_process_info()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/agent.py", line 2183, in get_supervisor_process_info
self.identifier = self.supervisor_client.supervisor.getIdentification()
File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/xmlrpc.py", line 460, in request
self.connection.request('POST', handler, request_body, self.headers)
File "/usr/lib64/python2.6/httplib.py", line 914, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
self.endheaders()
File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
self._send_output()
File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
self.send(msg)
File "/usr/lib64/python2.6/httplib.py", line 739, in send
self.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused
[20/Jun/2016 20:07:39 +0000] 7616 MainThread tmpfs INFO Reusing mounted tmpfs at /var/run/cloudera-scm-agent/process
[20/Jun/2016 20:07:39 +0000] 7616 MainThread agent INFO Deleting stale supervisor include /var/run/cloudera-scm-agent/supervisor/include/flood.conf.
[20/Jun/2016 20:07:40 +0000] 7616 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 1)
[20/Jun/2016 20:07:40 +0000] 7616 MainThread agent INFO Supervisor version: 3.0
[20/Jun/2016 20:07:40 +0000] 7616 MainThread agent INFO Successfully connected to supervisor
[20/Jun/2016 20:07:40 +0000] 7616 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes.
[20/Jun/2016 20:07:40 +0000] 7616 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes.
[20/Jun/2016 20:07:40 +0000] 7616 MainThread _cplogging INFO [20/Jun/2016:20:07:40] ENGINE Bus STARTING
[20/Jun/2016 20:07:40 +0000] 7616 MainThread _cplogging INFO [20/Jun/2016:20:07:40] ENGINE Started monitor thread '_TimeoutMonitor'.
[20/Jun/2016 20:07:40 +0000] 7616 MainThread _cplogging INFO [20/Jun/2016:20:07:40] ENGINE Serving on sg-master.dagupan.com:9000
[20/Jun/2016 20:07:40 +0000] 7616 MainThread _cplogging INFO [20/Jun/2016:20:07:40] ENGINE Bus STARTED
[20/Jun/2016 20:07:41 +0000] 7616 MainThread __init__ INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x2110a90>,)
[20/Jun/2016 20:07:41 +0000] 7616 MainThread agent INFO Setting default socket timeout to 30
[20/Jun/2016 20:07:41 +0000] 7616 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('HostMonitor',)
[20/Jun/2016 20:07:41 +0000] 7616 Monitor-HostMonitor network_interfaces INFO NIC iface eth0 doesn't support ETHTOOL (95)
[20/Jun/2016 20:07:41 +0000] 7616 Monitor-HostMonitor network_interfaces INFO NIC iface eth1 doesn't support ETHTOOL (95)
[20/Jun/2016 20:07:41 +0000] 7616 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.13 min:0.13 mean:0.13 max:0.13 LIFE_MAX:0.13
[20/Jun/2016 20:07:41 +0000] 7616 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
[20/Jun/2016 20:07:41 +0000] 7616 MainThread parcel INFO Agent does create users/groups and apply file permissions
[20/Jun/2016 20:07:41 +0000] 7616 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[20/Jun/2016 20:07:41 +0000] 7616 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[20/Jun/2016 20:07:41 +0000] 7616 MainThread agent INFO Flood daemon (re)start attempt
[20/Jun/2016 20:07:41 +0000] 7616 MainThread agent INFO Triggering supervisord update.
[20/Jun/2016 20:07:41 +0000] 7616 MainThread downloader ERROR Failed rack peer update: [Errno 111] Connection refused
[20/Jun/2016 20:07:41 +0000] 7616 MainThread agent INFO Active parcel list updated; recalculating component info.
[20/Jun/2016 20:07:42 +0000] 7616 MainThread throttling_logger INFO Identified java component java6 with full version JAVA_HOME=/usr/java/jdk1.6.0_31 java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) for requested version 6.
[20/Jun/2016 20:07:42 +0000] 7616 MainThread throttling_logger INFO Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.
[20/Jun/2016 20:08:11 +0000] 7616 DnsResolutionMonitor throttling_logger INFO Using java location: '/usr/java/jdk1.7.0_67-cloudera/bin/java'.