@bgooley ,   Yes, I enabled "Auto-TLS" during installation and I can't  find the way to disable.   I've uninstall and reinstall again without "Auto-TLS". It can run successfully now.   Thanks for your help!     ... View more

Hi @bgooley,   I've changed some settings, the error about "heartbeat"still ------------------------------------------------------------------------------------------------------------------------ MainThread agent ERROR Heartbeating to cfchadoopuat02.clg01.chailease.com.tw:7182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) SSLError: sslv3 alert bad certificate ------------------------------------------------------------------------------------------------------------------------   And the output when I enter "grep -v -e '^[[:space:]]*$' -e '^#' /etc/cloudera-scm-agent/config.ini": ------------------------------------------------------------------------------------------------------------------------ [General] server_host=cfchadoopuat02.clg01.chailease.com.tw server_port=7182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=INFO dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30 [Security] use_tls=1 max_cert_depth=9 [Hadoop] [Cloudera] [JDBC] [Cgroup_Paths] ------------------------------------------------------------------------------------------------------------------------   Thanks. ... View more

@bgooley Thanks for your reply.   Here is more information in the log files. The error about "heartbeat" appear in the last few lines. ------------------------------------------------------------------------------------------------------------------------ error: [Errno 111] Connection refused [17/Jul/2019 14:23:58 +0000] 7674 MainThread agent ERROR Failed to connect to previous supervisor. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 2019, in find_or_start_supervisor self.connect_to_supervisor() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 2099, in connect_to_supervisor self.supervisor_client = SupervisorWrapper.connect(self.cfg, self.os_ops) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/supervisor.py", line 136, in connect raise SupervisorError("Failed to connect to supervisor") SupervisorError: Failed to connect to supervisor [17/Jul/2019 14:23:58 +0000] 7674 MainThread main ERROR Top-level exception: Failed to connect to supervisor Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/main.py", line 107, in main_impl ag.start(legacy_supervisor) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 825, in start self.find_or_start_supervisor(enable_supervisor_start) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 2019, in find_or_start_supervisor self.connect_to_supervisor() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 2099, in connect_to_supervisor self.supervisor_client = SupervisorWrapper.connect(self.cfg, self.os_ops) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/supervisor.py", line 136, in connect raise SupervisorError("Failed to connect to supervisor") SupervisorError: Failed to connect to supervisor [17/Jul/2019 14:23:58 +0000] 7674 Dummy-1 agent INFO Stopping agent... [17/Jul/2019 14:23:58 +0000] 7674 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO ================================================================================ [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO SCM Agent Version: 6.2.0 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Agent Protocol Version: 4 [17/Jul/2019 14:24:11 +0000] 8512 MainThread __init__ INFO Agent UUID file was last modified at 2019-07-16 15:25:07.537834 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Using Host ID: f24982c8-1134-4803-8d55-9040914697b2 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Using directory: /run/cloudera-scm-agent [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Using supervisor binary path: /opt/cloudera/cm-agent/bin/../bin/supervisord [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Agent Logging Level: INFO [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Agent config: [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Security.use_tls = 1 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Security.max_cert_depth = 9 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.orphan_process_dir_staleness_threshold = 5184000 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.dns_resolution_collection_timeout_seconds = 30 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.orphan_process_dir_refresh_interval = 3600 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.local_filesystem_whitelist = ext2,ext3,ext4,xfs [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.stacks_log_bundle_max_bytes = 1073741824 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.metrics_url_timeout_seconds = 30.0 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.max_collection_wait_seconds = 10.0 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.task_metrics_timeout_seconds = 5.0 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.stacks_log_max_uncompressed_file_size_bytes = 5242880 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.scm_debug = INFO [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.server_host = cm.example.com [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.monitored_nodev_filesystem_types = nfs,nfs4,tmpfs [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.server_port = 7182 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.dns_resolution_collection_interval_seconds = 60 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO General.impala_profile_bundle_max_bytes = 1073741824 [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation. [17/Jul/2019 14:24:11 +0000] 8512 MainThread agent INFO Agent starting as pid 8512 user root(0) group root(0). [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found existing subsystem blkio at /sys/fs/cgroup/blkio [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found existing subsystem memory at /sys/fs/cgroup/memory [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found existing subsystem devices at /sys/fs/cgroup/devices [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found existing subsystem cpuacct at /sys/fs/cgroup/cpu,cpuacct [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found existing subsystem cpu at /sys/fs/cgroup/cpu,cpuacct [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found cgroups subsystem: cpu [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Adding more caps: {'writable_cgroup_dot_procs': True, 'default_cpu_shares': 1024, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True} [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found cgroups subsystem: cpuacct [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Adding more caps: {'has_cpuacct': True} [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found cgroups subsystem: memory [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Adding more caps: {'has_memory': True, 'default_memory_limit_in_bytes': 8796093022207, 'default_memory_soft_limit_in_bytes': 8796093022207} [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found cgroups subsystem: devices [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Adding more caps: {'has_devices': True} [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Found cgroups subsystem: blkio [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Adding more caps: {'has_blkio': True, 'default_blkio_weight': 1000} [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Ignoring cgroup avro_cap: default_memory_limit_in_bytes [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Ignoring cgroup avro_cap: default_blkio_weight [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Ignoring cgroup avro_cap: writable_cgroup_dot_procs [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Ignoring cgroup avro_cap: default_cpu_rt_runtime_us [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Ignoring cgroup avro_cap: default_memory_soft_limit_in_bytes [17/Jul/2019 14:24:13 +0000] 8512 MainThread cgroups INFO Ignoring cgroup avro_cap: default_cpu_shares [17/Jul/2019 14:24:13 +0000] 8512 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': 8796093022207, 'default_blkio_weight': 1000, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_memory_soft_limit_in_bytes': 8796093022207, 'has_cpuacct': True, 'default_cpu_shares': 1024, 'has_devices': True, 'has_blkio': True} [17/Jul/2019 14:24:13 +0000] 8512 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] [17/Jul/2019 14:24:13 +0000] 8512 MainThread filesystem_map INFO Using timeout of 2.000000 [17/Jul/2019 14:24:13 +0000] 8512 MainThread filesystem_map INFO Using join timeout of 0.100000 [17/Jul/2019 14:24:13 +0000] 8512 MainThread filesystem_map INFO Using tolerance of 60.000000 [17/Jul/2019 14:24:13 +0000] 8512 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4', 'xfs'] [17/Jul/2019 14:24:13 +0000] 8512 MainThread kt_renewer INFO Agent wide credential cache set to /var/run/cloudera-scm-agent/krb5cc_cm_agent_0 [17/Jul/2019 14:24:13 +0000] 8512 MainThread agent INFO Setting up supervisord event monitor. [17/Jul/2019 14:24:13 +0000] 8512 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 [17/Jul/2019 14:24:13 +0000] 8512 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 [17/Jul/2019 14:24:13 +0000] 8512 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 [17/Jul/2019 14:24:13 +0000] 8512 MainThread metrics INFO Importing tasktracker metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/tasktracker/schema.json [17/Jul/2019 14:24:13 +0000] 8512 MainThread ntp_monitor INFO Using timeout of 2.000000 [17/Jul/2019 14:24:13 +0000] 8512 MainThread dns_names INFO Using timeout of 30.000000 [17/Jul/2019 14:24:13 +0000] 8512 MainThread __init__ INFO Created DNS monitor. [17/Jul/2019 14:24:13 +0000] 8512 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 [17/Jul/2019 14:24:13 +0000] 8512 MainThread daemon INFO Importing metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/schema.json [17/Jul/2019 14:24:13 +0000] 8512 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CMF_PACKAGE_DIR': '/opt/cloudera/cm-agent/service', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_PARQUET_HOME': '/usr/lib/parquet', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/opt/cloudera/cm', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_SERVER_ROOT': '/opt/cloudera/cm', 'CDH_KUDU_HOME': '/usr/lib/kudu', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_HUE_HOME': '/usr/lib/hue', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} [17/Jul/2019 14:24:13 +0000] 8512 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. [17/Jul/2019 14:24:15 +0000] 8512 MainThread supervisor INFO Trying to connect to supervisor (Attempt 1) [17/Jul/2019 14:24:15 +0000] 8512 MainThread supervisor INFO Supervisor version: 3.0, pid: 8511 [17/Jul/2019 14:24:15 +0000] 8512 MainThread supervisor INFO Successfully connected to supervisor [17/Jul/2019 14:24:15 +0000] 8512 MainThread agent INFO Supervisor version: 3.0, pid: 8511 [17/Jul/2019 14:24:15 +0000] 8512 MainThread agent INFO Connecting to previous supervisor: agent-8511-1563344651. [17/Jul/2019 14:24:17 +0000] 8512 MainThread supervisor INFO Triggering supervisord update. [17/Jul/2019 14:24:17 +0000] 8512 MainThread _cplogging INFO [17/Jul/2019:14:24:17] ENGINE Bus STARTING [17/Jul/2019 14:24:17 +0000] 8512 MainThread _cplogging INFO [17/Jul/2019:14:24:17] ENGINE Started monitor thread '_TimeoutMonitor'. [17/Jul/2019 14:24:17 +0000] 8512 MainThread _cplogging INFO [17/Jul/2019:14:24:17] ENGINE Serving on http://127.0.0.1:9001 [17/Jul/2019 14:24:17 +0000] 8512 MainThread _cplogging INFO [17/Jul/2019:14:24:17] ENGINE Bus STARTED [17/Jul/2019 14:24:17 +0000] 8512 MainThread daemon INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x7fd829b574d0>,) [17/Jul/2019 14:24:17 +0000] 8512 MonitorDaemon-Scheduler daemon INFO Monitor ready to report: ('HostMonitor',) [17/Jul/2019 14:24:17 +0000] 8512 MainThread agent INFO Setting default socket timeout to 45 [17/Jul/2019 14:24:17 +0000] 8512 MainThread agent INFO Failed to read available parcel file: [Errno 2] No such file or directory: '/var/lib/cloudera-scm-agent/active_parcels.json' [17/Jul/2019 14:24:17 +0000] 8512 MainThread agent ERROR Heartbeating to cm.example.com:7182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) SSLError: sslv3 alert bad certificate [17/Jul/2019 14:24:17 +0000] 8512 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.01 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.01 ------------------------------------------------------------------------------------------------------------------------   And below is the output when I enter "grep -v -e '^[[:space:]]*$' -e '^#' /etc/cloudera-scm-agent/config.ini" ------------------------------------------------------------------------------------------------------------------------ [General] server_host=cm.example.com server_port=7182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=INFO dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30 [Security] use_tls=0 max_cert_depth=9 [Hadoop] [Cloudera] [JDBC] [Cgroup_Paths] ------------------------------------------------------------------------------------------------------------------------   Thanks! ... View more