Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CM Install on EC2 Failing at Agent Heartbeat Check

Highlighted

CM Install on EC2 Failing at Agent Heartbeat Check

New Contributor

I am attempting to set up a single node Cloudera Manager install on an EC2 instance in a private VPC (currently wide open to inbound traffic).  Everything in the installation seems to go fine until the heartbeat check from the agent.  Following are the tail details from the run failure (stripped of actual host name numbers, though note the two forms, ec2- and ip-):

 

Spoiler

Starting Cloudera Manager Agent...
server_host=ec2-##-###-###-##.us-west-2.compute.amazonaws.com
BEGIN sudo systemctl status cloudera-scm-agent
* cloudera-scm-agent.service - Cloudera Manager Agent Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2020-01-16 20:17:52 UTC; 17min ago
Main PID: 12937 (cmagent)
CGroup: /system.slice/cloudera-scm-agent.service
`-12937 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent

Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: self.run()
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: self._target(*self._args, **self._kwargs)
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/filesystem_map.py", line 27, in disk_usage_wrapper
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: usage = psutil.disk_usage(p)
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/__init__.py", line 1947, in disk_usage
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: return _psplatform.disk_usage(path)
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/_psposix.py", line 131, in disk_usage
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: st = os.statvfs(path)
Jan 16 20:27:39 ip-##-#-###-##.us-west-2.compute.internal cm[12937]: OSError: [Errno 2] No such file or directory: '/run/user/1000'
END (0)
BEGIN sudo systemctl restart cloudera-scm-agent
END (0)
agent logs:
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/'
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.out' for reading: No such file or directory
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.out' for reading: No such file or directory
END (0)
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/'
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread metrics INFO Importing tasktracker metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/tasktracker/schema.json
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread ntp_monitor INFO Using timeout of 2.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread dns_names INFO Using timeout of 30.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread __init__ INFO Created DNS monitor.
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread daemon INFO Importing metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/schema.json
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CMF_PACKAGE_DIR': '/opt/cloudera/cm-agent/service', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_PARQUET_HOME': '/usr/lib/parquet', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/opt/cloudera/cm', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_SERVER_ROOT': '/opt/cloudera/cm', 'CDH_KUDU_HOME': '/usr/lib/kudu', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_HUE_HOME': '/usr/lib/hue', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread supervisor INFO Trying to connect to supervisor (Attempt 1)
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread supervisor INFO Supervisor version: 3.0, pid: 5728
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread supervisor INFO Successfully connected to supervisor
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread agent INFO Supervisor version: 3.0, pid: 5728
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread agent INFO Connecting to previous supervisor: agent-5728-1579201834.
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread supervisor INFO Triggering supervisord update.
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:17:57] ENGINE Bus STARTING
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:17:57] ENGINE Started monitor thread '_TimeoutMonitor'.
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:17:57] ENGINE Serving on http://127.0.0.1:9001
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:17:57] ENGINE Bus STARTED
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread daemon INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x7ff5727a6fd0>,)
>>[16/Jan/2020 20:17:59 +0000] 12937 MonitorDaemon-Scheduler daemon INFO Monitor ready to report: ('HostMonitor',)
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Setting default socket timeout to 45
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Previously active parcels: {}
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.03 min:0.03 mean:0.03 max:0.03 LIFE_MAX:0.03
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO CM server guid: f09ffbce-42ef-4fe0-b1d7-69a099ec5379
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread parcel INFO Agent does create users/groups and apply file permissions
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread throttling_logger WARNING Failed parsing alternatives line: libnssckbi.so.x86_64 string index out of range link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Flood daemon (re)start attempt
>>[16/Jan/2020 20:18:02 +0000] 12937 MainThread throttling_logger INFO Identified java component java8 with full version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) for requested version .
>>[16/Jan/2020 20:27:41 +0000] 12937 Monitor-HostMonitor throttling_logger ERROR Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/user/1000
>>[16/Jan/2020 20:28:02 +0000] 12937 MainThread heartbeat_tracker INFO HB stats (seconds): num:41 LIFE_MIN:0.01 min:0.00 mean:0.01 max:0.02 LIFE_MAX:0.03
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread agent INFO Stopping agent...
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPING
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 9001)) shut down
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Stopped thread '_TimeoutMonitor'.
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPED
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPING
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 9001)) already shut down
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE No thread running for None.
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPED
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus EXITING
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus EXITED
>>[16/Jan/2020 20:35:07 +0000] 12937 MainThread agent INFO Agent exiting; caught signal 15
END (0)
end of agent logs.
scm agent restarted
Installation script completed successfully.
all done
closing logging file descriptor
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread metrics INFO Importing tasktracker metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/tasktracker/schema.json
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread ntp_monitor INFO Using timeout of 2.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread dns_names INFO Using timeout of 30.000000
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread __init__ INFO Created DNS monitor.
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread daemon INFO Importing metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/schema.json
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CMF_PACKAGE_DIR': '/opt/cloudera/cm-agent/service', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_PARQUET_HOME': '/usr/lib/parquet', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/opt/cloudera/cm', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_SERVER_ROOT': '/opt/cloudera/cm', 'CDH_KUDU_HOME': '/usr/lib/kudu', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_HUE_HOME': '/usr/lib/hue', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
>>[16/Jan/2020 20:17:53 +0000] 12937 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread supervisor INFO Trying to connect to supervisor (Attempt 1)
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread supervisor INFO Supervisor version: 3.0, pid: 5728
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread supervisor INFO Successfully connected to supervisor
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread agent INFO Supervisor version: 3.0, pid: 5728
>>[16/Jan/2020 20:17:55 +0000] 12937 MainThread agent INFO Connecting to previous supervisor: agent-5728-1579201834.
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread supervisor INFO Triggering supervisord update.
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:17:57] ENGINE Bus STARTING
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:17:57] ENGINE Started monitor thread '_TimeoutMonitor'.
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:17:57] ENGINE Serving on http://127.0.0.1:9001
>>[16/Jan/2020 20:17:57 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:17:57] ENGINE Bus STARTED
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread daemon INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x7ff5727a6fd0>,)
>>[16/Jan/2020 20:17:59 +0000] 12937 MonitorDaemon-Scheduler daemon INFO Monitor ready to report: ('HostMonitor',)
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Setting default socket timeout to 45
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Previously active parcels: {}
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.03 min:0.03 mean:0.03 max:0.03 LIFE_MAX:0.03
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO CM server guid: f09ffbce-42ef-4fe0-b1d7-69a099ec5379
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread parcel INFO Agent does create users/groups and apply file permissions
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread throttling_logger WARNING Failed parsing alternatives line: libnssckbi.so.x86_64 string index out of range link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
>>[16/Jan/2020 20:17:59 +0000] 12937 MainThread agent INFO Flood daemon (re)start attempt
>>[16/Jan/2020 20:18:02 +0000] 12937 MainThread throttling_logger INFO Identified java component java8 with full version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) for requested version .
>>[16/Jan/2020 20:27:41 +0000] 12937 Monitor-HostMonitor throttling_logger ERROR Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/user/1000
>>[16/Jan/2020 20:28:02 +0000] 12937 MainThread heartbeat_tracker INFO HB stats (seconds): num:41 LIFE_MIN:0.01 min:0.00 mean:0.01 max:0.02 LIFE_MAX:0.03
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread agent INFO Stopping agent...
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPING
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 9001)) shut down
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Stopped thread '_TimeoutMonitor'.
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPED
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPING
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 9001)) already shut down
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE No thread running for None.
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPED
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus EXITING
>>[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus EXITED
>>[16/Jan/2020 20:35:07 +0000] 12937 MainThread agent INFO Agent exiting; caught signal 15

 

And here is the cloudera-scm-agent.log:

 

Spoiler

[16/Jan/2020 20:27:41 +0000] 12937 Monitor-HostMonitor throttling_logger ERROR Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/user/1000
[16/Jan/2020 20:28:02 +0000] 12937 MainThread heartbeat_tracker INFO HB stats (seconds): num:41 LIFE_MIN:0.01 min:0.00 mean:0.01 max:0.02 LIFE_MAX:0.03
[16/Jan/2020 20:35:06 +0000] 12937 MainThread agent INFO Stopping agent...
[16/Jan/2020 20:35:06 +0000] 12937 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPING
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 9001)) shut down
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Stopped thread '_TimeoutMonitor'.
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPED
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPING
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 9001)) already shut down
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE No thread running for None.
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus STOPPED
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus EXITING
[16/Jan/2020 20:35:06 +0000] 12937 MainThread _cplogging INFO [16/Jan/2020:20:35:06] ENGINE Bus EXITED
[16/Jan/2020 20:35:07 +0000] 12937 MainThread agent INFO Agent exiting; caught signal 15
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO ================================================================================
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO SCM Agent Version: 6.3.0
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Agent Protocol Version: 4
[16/Jan/2020 20:35:07 +0000] 14484 MainThread __init__ INFO Agent UUID file was last modified at 2020-01-16 19:10:34.135133
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Using Host ID: b86fe43c-3b46-4b90-a8bf-866200c3da65
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Using directory: /run/cloudera-scm-agent
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Using supervisor binary path: /opt/cloudera/cm-agent/bin/../bin/supervisord
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Agent Logging Level: INFO
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Agent config:
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Security.use_tls = 0
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Security.max_cert_depth = 9
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.orphan_process_dir_staleness_threshold = 5184000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.dns_resolution_collection_timeout_seconds = 30
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.orphan_process_dir_refresh_interval = 3600
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.local_filesystem_whitelist = ext2,ext3,ext4,xfs
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.stacks_log_bundle_max_bytes = 1073741824
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.metrics_url_timeout_seconds = 30.0
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.max_collection_wait_seconds = 10.0
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.task_metrics_timeout_seconds = 5.0
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.stacks_log_max_uncompressed_file_size_bytes = 5242880
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.scm_debug = INFO
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.server_host = ec2-##-###-###-##.us-west-2.compute.amazonaws.com
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.monitored_nodev_filesystem_types = nfs,nfs4,tmpfs
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.server_port = 7182
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO General.dns_resolution_collection_interval_seconds = 60
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Agent starting as pid 14484 user root(0) group root(0).
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found existing subsystem cpuacct at /sys/fs/cgroup/cpu,cpuacct
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found existing subsystem cpu at /sys/fs/cgroup/cpu,cpuacct
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found existing subsystem devices at /sys/fs/cgroup/devices
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found existing subsystem memory at /sys/fs/cgroup/memory
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found existing subsystem blkio at /sys/fs/cgroup/blkio
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found cgroups subsystem: cpu
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Adding more caps: {'writable_cgroup_dot_procs': True, 'default_cpu_shares': 1024, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True}
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found cgroups subsystem: cpuacct
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Adding more caps: {'has_cpuacct': True}
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found cgroups subsystem: memory
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Adding more caps: {'has_memory': True, 'default_memory_limit_in_bytes': 8796093022207, 'default_memory_soft_limit_in_bytes': 8796093022207}
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found cgroups subsystem: devices
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Adding more caps: {'has_devices': True}
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Found cgroups subsystem: blkio
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Adding more caps: {'has_blkio': True, 'default_blkio_weight': 1000}
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Ignoring cgroup avro_cap: default_memory_limit_in_bytes
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Ignoring cgroup avro_cap: default_blkio_weight
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Ignoring cgroup avro_cap: writable_cgroup_dot_procs
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Ignoring cgroup avro_cap: default_cpu_rt_runtime_us
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Ignoring cgroup avro_cap: default_memory_soft_limit_in_bytes
[16/Jan/2020 20:35:07 +0000] 14484 MainThread cgroups INFO Ignoring cgroup avro_cap: default_cpu_shares
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': 8796093022207, 'default_blkio_weight': 1000, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_memory_soft_limit_in_bytes': 8796093022207, 'has_cpuacct': True, 'default_cpu_shares': 1024, 'has_devices': True, 'has_blkio': True}
[16/Jan/2020 20:35:07 +0000] 14484 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[16/Jan/2020 20:35:07 +0000] 14484 MainThread filesystem_map INFO Using timeout of 2.000000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread filesystem_map INFO Using join timeout of 0.100000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread filesystem_map INFO Using tolerance of 60.000000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4', 'xfs']
[16/Jan/2020 20:35:07 +0000] 14484 MainThread kt_renewer INFO Agent wide credential cache set to /var/run/cloudera-scm-agent/krb5cc_cm_agent_0
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Setting up supervisord event monitor.
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread metrics INFO Importing tasktracker metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/tasktracker/schema.json
[16/Jan/2020 20:35:07 +0000] 14484 MainThread ntp_monitor INFO Using timeout of 2.000000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread dns_names INFO Using timeout of 30.000000
[16/Jan/2020 20:35:07 +0000] 14484 MainThread __init__ INFO Created DNS monitor.
[16/Jan/2020 20:35:07 +0000] 14484 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
[16/Jan/2020 20:35:07 +0000] 14484 MainThread daemon INFO Importing metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/schema.json
[16/Jan/2020 20:35:08 +0000] 14484 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CMF_PACKAGE_DIR': '/opt/cloudera/cm-agent/service', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_PARQUET_HOME': '/usr/lib/parquet', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/opt/cloudera/cm', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_SERVER_ROOT': '/opt/cloudera/cm', 'CDH_KUDU_HOME': '/usr/lib/kudu', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_HUE_HOME': '/usr/lib/hue', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[16/Jan/2020 20:35:08 +0000] 14484 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[16/Jan/2020 20:35:10 +0000] 14484 MainThread supervisor INFO Trying to connect to supervisor (Attempt 1)
[16/Jan/2020 20:35:10 +0000] 14484 MainThread supervisor INFO Supervisor version: 3.0, pid: 5728
[16/Jan/2020 20:35:10 +0000] 14484 MainThread supervisor INFO Successfully connected to supervisor
[16/Jan/2020 20:35:10 +0000] 14484 MainThread agent INFO Supervisor version: 3.0, pid: 5728
[16/Jan/2020 20:35:10 +0000] 14484 MainThread agent INFO Connecting to previous supervisor: agent-5728-1579201834.
[16/Jan/2020 20:35:12 +0000] 14484 MainThread supervisor INFO Triggering supervisord update.
[16/Jan/2020 20:35:12 +0000] 14484 MainThread _cplogging INFO [16/Jan/2020:20:35:12] ENGINE Bus STARTING
[16/Jan/2020 20:35:12 +0000] 14484 MainThread _cplogging INFO [16/Jan/2020:20:35:12] ENGINE Started monitor thread '_TimeoutMonitor'.
[16/Jan/2020 20:35:12 +0000] 14484 MainThread _cplogging INFO [16/Jan/2020:20:35:12] ENGINE Serving on http://127.0.0.1:9001
[16/Jan/2020 20:35:12 +0000] 14484 MainThread _cplogging INFO [16/Jan/2020:20:35:12] ENGINE Bus STARTED
[16/Jan/2020 20:35:13 +0000] 14484 MainThread daemon INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x7f3ad475dfd0>,)
[16/Jan/2020 20:35:13 +0000] 14484 MainThread agent INFO Setting default socket timeout to 45
[16/Jan/2020 20:35:13 +0000] 14484 MonitorDaemon-Scheduler daemon INFO Monitor ready to report: ('HostMonitor',)
[16/Jan/2020 20:35:13 +0000] 14484 MainThread agent INFO Previously active parcels: {}
[16/Jan/2020 20:35:13 +0000] 14484 MainThread agent INFO Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro
[16/Jan/2020 20:35:13 +0000] 14484 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.05 min:0.05 mean:0.05 max:0.05 LIFE_MAX:0.05
[16/Jan/2020 20:35:13 +0000] 14484 MainThread agent INFO CM server guid: f09ffbce-42ef-4fe0-b1d7-69a099ec5379
[16/Jan/2020 20:35:13 +0000] 14484 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
[16/Jan/2020 20:35:13 +0000] 14484 MainThread parcel INFO Agent does create users/groups and apply file permissions
[16/Jan/2020 20:35:13 +0000] 14484 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[16/Jan/2020 20:35:13 +0000] 14484 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[16/Jan/2020 20:35:13 +0000] 14484 MainThread throttling_logger WARNING Failed parsing alternatives line: libnssckbi.so.x86_64 string index out of range link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
[16/Jan/2020 20:35:14 +0000] 14484 MainThread agent INFO Flood daemon (re)start attempt
[16/Jan/2020 20:35:16 +0000] 14484 MainThread throttling_logger INFO Identified java component java8 with full version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) for requested version .
[16/Jan/2020 20:45:16 +0000] 14484 MainThread heartbeat_tracker INFO HB stats (seconds): num:41 LIFE_MIN:0.00 min:0.00 mean:0.01 max:0.02 LIFE_MAX:0.05

 

Thanks in advance.

4 REPLIES 4

Re: CM Install on EC2 Failing at Agent Heartbeat Check

Community Manager

Hi @sunwugong ,

 

The error message: OSError: [Errno 2] No such file or directory: '/run/user/1000' seems to point out that you may have some mount point and later on umounted them. However, Cloudera Manager agent still tries to grab the filesystem information of that mount point and fails.


To temporarily suppress this error, update the /etc/cloudera-scm-agent/config.ini on each of the hosts affected to:

In the line that lists the nodev filesystems:
# The list of non-device (nodev) filesystem types which will be monitored.

monitored_nodev_filesystem_types=nfs,nfs4,tmpfs


Please change the uncommented line to read as (remove tmpfs):

monitored_nodev_filesystem_types=nfs,nfs4


Restart Cloudera Manager agent with:

service cloudera-scm-agent restart

 

Thanks and hope this helps,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Re: CM Install on EC2 Failing at Agent Heartbeat Check

New Contributor

Hello, and thanks for your comment!  That particular error was new in my attempts to get this working so I actually went ahead and spun up a new EC2 instance and did a new install just to undo any new problems I may have created in trying to fix things.  I now have the very first set of logs for a failure to hear heartbeat (which does not include that user error):

 

This is from the Cloudera GUI (I obscured the IP, though oddly my IP stars "52." and in one case it looked like it used "152." visible as "1##." below):

Spoiler

/tmp/scm_prepare_node.3QUvYeoQ
using SSH_CLIENT to get the SCM hostname: ##.##.##.## 48496 22
opening logging file descriptor
Starting installation script...
Acquiring installation lock...
BEGIN flock 4
END (0)
Detecting root privileges...
effective UID is 1000
BEGIN which pbrun
which: no pbrun in (/usr/local/bin:/usr/bin)
END (1)
BEGIN sudo -S id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
END (0)
Using 'sudo ' to acquire root privileges
Detecting distribution...
BEGIN grep Tikanga /etc/redhat-release
END (1)
BEGIN grep 'Scientific Linux release 5' /etc/redhat-release
END (1)
BEGIN grep Santiago /etc/redhat-release
END (1)
BEGIN grep 'CentOS Linux release 6' /etc/redhat-release
END (1)
BEGIN grep 'CentOS release 6' /etc/redhat-release
END (1)
BEGIN grep 'Scientific Linux release 6' /etc/redhat-release
END (1)
BEGIN grep Maipo /etc/redhat-release
END (1)
BEGIN grep 'CentOS Linux release 7' /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
END (0)
/etc/redhat-release ==> CentOS 7
Detecting Cloudera Manager Server...
BEGIN host -t PTR ##.##.##.##
/tmp/scm_prepare_node.3QUvYeoQ/scm_prepare_node.sh: line 106: host: command not found
END (127)
BEGIN which python
END (0)
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' ##.##.##.## 7182
/usr/bin/python
END (0)
BEGIN which wget
/usr/bin/wget
END (0)
BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo

END (0)
Installing package repositories...
Checking https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/repodata/repomd.xml
Using baseurl = https://archive.cloudera.com/cm6/6.3.1/redhat7/yum
installing repository file /tmp/scm_prepare_node.3QUvYeoQ/repos/rhel7/cloudera-manager.repo
repository file /tmp/scm_prepare_node.3QUvYeoQ/repos/rhel7/cloudera-manager.repo installed
installing rpm keys
BEGIN gpg --import /tmp/scm_prepare_node.3QUvYeoQ/customGPG
gpg: keyring `/tmp/scm_prepare_node.3QUvYeoQ/gnupg.3kh7URBW57/secring.gpg' created
gpg: keyring `/tmp/scm_prepare_node.3QUvYeoQ/gnupg.3kh7URBW57/pubring.gpg' created
gpg: /tmp/scm_prepare_node.3QUvYeoQ/gnupg.3kh7URBW57/trustdb.gpg: trustdb created
gpg: key 02A818DD: public key "Cloudera Apt Repository" imported
gpg: key E8F86ACD: public key "Yum Maintainer <webmaster@cloudera.com>" imported
gpg: key B0B19C9F: public key "Parameterized Build <security@cloudera.com>" imported
gpg: key 84415700: public key "Cloudera <security@cloudera.com>" imported
gpg: key 36F57F35: public key "Cloudera <security@cloudera.com>" imported
gpg: Total number processed: 5
gpg: imported: 5 (RSA: 3)
END (0)
BEGIN sudo rpm --import /tmp/scm_prepare_node.3QUvYeoQ/F36A89E33CC1BD0F71079007327574EE02A818DD.pub
END (0)
BEGIN sudo rpm --import /tmp/scm_prepare_node.3QUvYeoQ/5F14D39EF0681ACA6F044A43F90C0D8FE8F86ACD.pub
END (0)
BEGIN sudo rpm --import /tmp/scm_prepare_node.3QUvYeoQ/9543951160C284C0E7CA254573985D43B0B19C9F.pub
END (0)
BEGIN sudo rpm --import /tmp/scm_prepare_node.3QUvYeoQ/CECDB80C4E9004B0CFE852962279662784415700.pub
END (0)
BEGIN sudo rpm --import /tmp/scm_prepare_node.3QUvYeoQ/DF2C4DD7629B1AC08A0966E00F65552736F57F35.pub
END (0)
Refreshing package metadata...
BEGIN sudo yum --disablerepo=* --enablerepo=cloudera* clean all
Loaded plugins: fastestmirror
Cleaning repos: cloudera-manager
Cleaning up list of fastest mirrors
Other repos take up 73 M of disk space (use --verbose for details)
END (0)
BEGIN sudo rm -Rf /var/cache/yum/x86_64
END (0)
BEGIN sudo yum --disablerepo=* --enablerepo=cloudera* makecache
Loaded plugins: fastestmirror
Determining fastest mirrors
Metadata Cache Created
END (0)
Installing oracle-j2sdk1.8 package...
BEGIN sudo yum --disablerepo=* --enablerepo=cloudera* list installed oracle-j2sdk1.8
Loaded plugins: fastestmirror
Installed Packages
oracle-j2sdk1.8.x86_64 1.8.0+update181-1 @Cloudera-manager
END (0)
BEGIN echo oracle-j2sdk1.8 cloudera-manager-agent cloudera-manager-daemons | grep oracle-j2sdk1.8
oracle-j2sdk1.8 cloudera-manager-agent cloudera-manager-daemons
END (0)
BEGIN sudo yum --disablerepo=* --enablerepo=cloudera* info oracle-j2sdk1.8
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Installed Packages
Name : oracle-j2sdk1.8
Arch : x86_64
Version : 1.8.0+update181
Release : 1
Size : 364 M
Repo : installed
From repo : cloudera-manager
Summary : no description given
URL : http://example.com/no-uri-given
License : unknown
Description : no description given

END (0)
BEGIN sudo yum -y install oracle-j2sdk1.8.x86_64
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.web-ster.com
* extras: mirror.web-ster.com
* updates: mirror.web-ster.com
Package oracle-j2sdk1.8-1.8.0+update181-1.x86_64 already installed and latest version
Nothing to do
END (0)
remote package oracle-j2sdk1.8 installed
Installing cloudera-manager-agent package...
BEGIN sudo yum --disablerepo=* --enablerepo=cloudera* list installed cloudera-manager-agent
Loaded plugins: fastestmirror
Error: No matching Packages to list
END (1)
BEGIN sudo yum --disablerepo=* --enablerepo=cloudera* info cloudera-manager-agent
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Available Packages
Name : cloudera-manager-agent
Arch : x86_64
Version : 6.3.1
Release : 1466458.el7
Size : 10 M
Repo : cloudera-manager
Summary : The Cloudera Manager Agent
URL : http://www.cloudera.com
License : Proprietary
Description : The Cloudera Manager Agent.
:
: The Agent is deployed to machines running services managed by
: Cloudera Manager.

END (0)
Version : 6.3.1
Release : 1466458.el7
BEGIN sudo yum -y install cloudera-manager-agent
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.tocici.com
* extras: mirror.tocici.com
* updates: mirror.tocici.com
Resolving Dependencies
--> Running transaction check
---> Package cloudera-manager-agent.x86_64 0:6.3.1-1466458.el7 will be installed
--> Processing Dependency: libxslt for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: httpd for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: /lib/lsb/init-functions for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: fuse for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: cyrus-sasl-gssapi for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: openssl-devel for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: bind-utils for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: MySQL-python for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: fuse-libs for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: cyrus-sasl-plain for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: mod_ssl for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Processing Dependency: python-psycopg2 for package: cloudera-manager-agent-6.3.1-1466458.el7.x86_64
--> Running transaction check
---> Package MySQL-python.x86_64 0:1.2.5-1.el7 will be installed
---> Package bind-utils.x86_64 32:9.11.4-9.P2.el7 will be installed
--> Processing Dependency: bind-libs-lite(x86-64) = 32:9.11.4-9.P2.el7 for package: 32:bind-utils-9.11.4-9.P2.el7.x86_64
--> Processing Dependency: bind-libs(x86-64) = 32:9.11.4-9.P2.el7 for package: 32:bind-utils-9.11.4-9.P2.el7.x86_64
--> Processing Dependency: liblwres.so.160()(64bit) for package: 32:bind-utils-9.11.4-9.P2.el7.x86_64
--> Processing Dependency: libisccfg.so.160()(64bit) for package: 32:bind-utils-9.11.4-9.P2.el7.x86_64
--> Processing Dependency: libisc.so.169()(64bit) for package: 32:bind-utils-9.11.4-9.P2.el7.x86_64
--> Processing Dependency: libirs.so.160()(64bit) for package: 32:bind-utils-9.11.4-9.P2.el7.x86_64
--> Processing Dependency: libdns.so.1102()(64bit) for package: 32:bind-utils-9.11.4-9.P2.el7.x86_64
--> Processing Dependency: libbind9.so.160()(64bit) for package: 32:bind-utils-9.11.4-9.P2.el7.x86_64
---> Package cyrus-sasl-gssapi.x86_64 0:2.1.26-23.el7 will be installed
---> Package cyrus-sasl-plain.x86_64 0:2.1.26-23.el7 will be installed
---> Package fuse.x86_64 0:2.9.2-11.el7 will be installed
---> Package fuse-libs.x86_64 0:2.9.2-11.el7 will be installed
---> Package httpd.x86_64 0:2.4.6-90.el7.centos will be installed
--> Processing Dependency: httpd-tools = 2.4.6-90.el7.centos for package: httpd-2.4.6-90.el7.centos.x86_64
--> Processing Dependency: system-logos >= 7.92.1-1 for package: httpd-2.4.6-90.el7.centos.x86_64
--> Processing Dependency: /etc/mime.types for package: httpd-2.4.6-90.el7.centos.x86_64
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-90.el7.centos.x86_64
--> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-90.el7.centos.x86_64
---> Package libxslt.x86_64 0:1.1.28-5.el7 will be installed
---> Package mod_ssl.x86_64 1:2.4.6-90.el7.centos will be installed
---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed
--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
--> Processing Dependency: zlib-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
---> Package python-psycopg2.x86_64 0:2.5.1-4.el7 will be installed
---> Package redhat-lsb-core.x86_64 0:4.1-27.el7.centos.1 will be installed
--> Processing Dependency: redhat-lsb-submod-security(x86-64) = 4.1-27.el7.centos.1 for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: spax for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/bin/time for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/bin/patch for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/bin/m4 for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/bin/lpr for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/bin/lp for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/bin/bc for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/bin/batch for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/bin/at for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /bin/mailx for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /bin/ed for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Running transaction check
---> Package apr.x86_64 0:1.4.8-5.el7 will be installed
---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed
---> Package at.x86_64 0:3.1.13-24.el7 will be installed
---> Package bc.x86_64 0:1.06.95-13.el7 will be installed
---> Package bind-libs.x86_64 32:9.11.4-9.P2.el7 will be installed
--> Processing Dependency: bind-license = 32:9.11.4-9.P2.el7 for package: 32:bind-libs-9.11.4-9.P2.el7.x86_64
---> Package bind-libs-lite.x86_64 32:9.9.4-72.el7 will be updated
--> Processing Dependency: libdns-export.so.100()(64bit) for package: 12:dhclient-4.2.5-68.el7.centos.1.x86_64
--> Processing Dependency: libisc-export.so.95()(64bit) for package: 12:dhclient-4.2.5-68.el7.centos.1.x86_64
---> Package bind-libs-lite.x86_64 32:9.11.4-9.P2.el7 will be an update
---> Package centos-logos.noarch 0:70.0.6-3.el7.centos will be installed
---> Package cups-client.x86_64 1:1.6.3-40.el7 will be installed
--> Processing Dependency: cups-libs(x86-64) = 1:1.6.3-40.el7 for package: 1:cups-client-1.6.3-40.el7.x86_64
--> Processing Dependency: libcups.so.2()(64bit) for package: 1:cups-client-1.6.3-40.el7.x86_64
--> Processing Dependency: libavahi-common.so.3()(64bit) for package: 1:cups-client-1.6.3-40.el7.x86_64
--> Processing Dependency: libavahi-client.so.3()(64bit) for package: 1:cups-client-1.6.3-40.el7.x86_64
---> Package ed.x86_64 0:1.9-4.el7 will be installed
---> Package httpd-tools.x86_64 0:2.4.6-90.el7.centos will be installed
---> Package krb5-devel.x86_64 0:1.15.1-37.el7_7.2 will be installed
--> Processing Dependency: libkadm5(x86-64) = 1.15.1-37.el7_7.2 for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: krb5-libs(x86-64) = 1.15.1-37.el7_7.2 for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
---> Package m4.x86_64 0:1.4.16-10.el7 will be installed
---> Package mailcap.noarch 0:2.1.41-2.el7 will be installed
---> Package mailx.x86_64 0:12.5-19.el7 will be installed
---> Package openssl-libs.x86_64 1:1.0.2k-16.el7 will be updated
--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-16.el7 for package: 1:openssl-1.0.2k-16.el7.x86_64
---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be an update
---> Package patch.x86_64 0:2.7.1-12.el7_7 will be installed
---> Package redhat-lsb-submod-security.x86_64 0:4.1-27.el7.centos.1 will be installed
---> Package spax.x86_64 0:1.5.2-13.el7 will be installed
---> Package time.x86_64 0:1.7-45.el7 will be installed
---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed
--> Running transaction check
---> Package avahi-libs.x86_64 0:0.6.31-19.el7 will be installed
---> Package bind-license.noarch 32:9.9.4-72.el7 will be updated
---> Package bind-license.noarch 32:9.11.4-9.P2.el7 will be an update
---> Package cups-libs.x86_64 1:1.6.3-40.el7 will be installed
---> Package dhclient.x86_64 12:4.2.5-68.el7.centos.1 will be updated
---> Package dhclient.x86_64 12:4.2.5-77.el7.centos will be an update
--> Processing Dependency: dhcp-libs(x86-64) = 12:4.2.5-77.el7.centos for package: 12:dhclient-4.2.5-77.el7.centos.x86_64
--> Processing Dependency: dhcp-common = 12:4.2.5-77.el7.centos for package: 12:dhclient-4.2.5-77.el7.centos.x86_64
--> Processing Dependency: libisc-export.so.169()(64bit) for package: 12:dhclient-4.2.5-77.el7.centos.x86_64
--> Processing Dependency: libdns-export.so.1102()(64bit) for package: 12:dhclient-4.2.5-77.el7.centos.x86_64
---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed
---> Package krb5-libs.x86_64 0:1.15.1-34.el7 will be updated
---> Package krb5-libs.x86_64 0:1.15.1-37.el7_7.2 will be an update
---> Package libcom_err-devel.x86_64 0:1.42.9-16.el7 will be installed
--> Processing Dependency: libcom_err(x86-64) = 1.42.9-16.el7 for package: libcom_err-devel-1.42.9-16.el7.x86_64
---> Package libkadm5.x86_64 0:1.15.1-37.el7_7.2 will be installed
---> Package libselinux-devel.x86_64 0:2.5-14.1.el7 will be installed
--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-14.1.el7.x86_64
--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-14.1.el7.x86_64
--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-14.1.el7.x86_64
---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed
---> Package openssl.x86_64 1:1.0.2k-16.el7 will be updated
---> Package openssl.x86_64 1:1.0.2k-19.el7 will be an update
--> Running transaction check
---> Package bind-export-libs.x86_64 32:9.11.4-9.P2.el7 will be installed
---> Package dhcp-common.x86_64 12:4.2.5-68.el7.centos.1 will be updated
---> Package dhcp-common.x86_64 12:4.2.5-77.el7.centos will be an update
---> Package dhcp-libs.x86_64 12:4.2.5-68.el7.centos.1 will be updated
---> Package dhcp-libs.x86_64 12:4.2.5-77.el7.centos will be an update
---> Package libcom_err.x86_64 0:1.42.9-13.el7 will be updated
--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: e2fsprogs-1.42.9-13.el7.x86_64
--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: e2fsprogs-libs-1.42.9-13.el7.x86_64
--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: libss-1.42.9-13.el7.x86_64
---> Package libcom_err.x86_64 0:1.42.9-16.el7 will be an update
---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed
---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed
--> Running transaction check
---> Package e2fsprogs.x86_64 0:1.42.9-13.el7 will be updated
---> Package e2fsprogs.x86_64 0:1.42.9-16.el7 will be an update
---> Package e2fsprogs-libs.x86_64 0:1.42.9-13.el7 will be updated
---> Package e2fsprogs-libs.x86_64 0:1.42.9-16.el7 will be an update
---> Package libss.x86_64 0:1.42.9-13.el7 will be updated
---> Package libss.x86_64 0:1.42.9-16.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
cloudera-manager-agent x86_64 6.3.1-1466458.el7 cloudera-manager 10 M
Installing for dependencies:
MySQL-python x86_64 1.2.5-1.el7 base 90 k
apr x86_64 1.4.8-5.el7 base 103 k
apr-util x86_64 1.5.2-6.el7 base 92 k
at x86_64 3.1.13-24.el7 base 51 k
avahi-libs x86_64 0.6.31-19.el7 base 61 k
bc x86_64 1.06.95-13.el7 base 115 k
bind-export-libs x86_64 32:9.11.4-9.P2.el7 base 1.1 M
bind-libs x86_64 32:9.11.4-9.P2.el7 base 154 k
bind-utils x86_64 32:9.11.4-9.P2.el7 base 258 k
centos-logos noarch 70.0.6-3.el7.centos base 21 M
cups-client x86_64 1:1.6.3-40.el7 base 151 k
cups-libs x86_64 1:1.6.3-40.el7 base 358 k
cyrus-sasl-gssapi x86_64 2.1.26-23.el7 base 41 k
cyrus-sasl-plain x86_64 2.1.26-23.el7 base 39 k
ed x86_64 1.9-4.el7 base 72 k
fuse x86_64 2.9.2-11.el7 base 86 k
fuse-libs x86_64 2.9.2-11.el7 base 93 k
httpd x86_64 2.4.6-90.el7.centos base 2.7 M
httpd-tools x86_64 2.4.6-90.el7.centos base 91 k
keyutils-libs-devel x86_64 1.5.8-3.el7 base 37 k
krb5-devel x86_64 1.15.1-37.el7_7.2 updates 272 k
libcom_err-devel x86_64 1.42.9-16.el7 base 32 k
libkadm5 x86_64 1.15.1-37.el7_7.2 updates 178 k
libselinux-devel x86_64 2.5-14.1.el7 base 187 k
libsepol-devel x86_64 2.5-10.el7 base 77 k
libverto-devel x86_64 0.2.5-4.el7 base 12 k
libxslt x86_64 1.1.28-5.el7 base 242 k
m4 x86_64 1.4.16-10.el7 base 256 k
mailcap noarch 2.1.41-2.el7 base 31 k
mailx x86_64 12.5-19.el7 base 245 k
mod_ssl x86_64 1:2.4.6-90.el7.centos base 112 k
openssl-devel x86_64 1:1.0.2k-19.el7 base 1.5 M
patch x86_64 2.7.1-12.el7_7 updates 111 k
pcre-devel x86_64 8.32-17.el7 base 480 k
python-psycopg2 x86_64 2.5.1-4.el7 base 132 k
redhat-lsb-core x86_64 4.1-27.el7.centos.1 base 38 k
redhat-lsb-submod-security
x86_64 4.1-27.el7.centos.1 base 15 k
spax x86_64 1.5.2-13.el7 base 260 k
time x86_64 1.7-45.el7 base 30 k
zlib-devel x86_64 1.2.7-18.el7 base 50 k
Updating for dependencies:
bind-libs-lite x86_64 32:9.11.4-9.P2.el7 base 1.1 M
bind-license noarch 32:9.11.4-9.P2.el7 base 88 k
dhclient x86_64 12:4.2.5-77.el7.centos base 285 k
dhcp-common x86_64 12:4.2.5-77.el7.centos base 176 k
dhcp-libs x86_64 12:4.2.5-77.el7.centos base 133 k
e2fsprogs x86_64 1.42.9-16.el7 base 700 k
e2fsprogs-libs x86_64 1.42.9-16.el7 base 167 k
krb5-libs x86_64 1.15.1-37.el7_7.2 updates 805 k
libcom_err x86_64 1.42.9-16.el7 base 41 k
libss x86_64 1.42.9-16.el7 base 46 k
openssl x86_64 1:1.0.2k-19.el7 base 493 k
openssl-libs x86_64 1:1.0.2k-19.el7 base 1.2 M

Transaction Summary
================================================================================
Install 1 Package (+40 Dependent packages)
Upgrade ( 12 Dependent packages)

Total download size: 46 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
http://mirror.tocici.com/centos/7.7.1908/updates/x86_64/Packages/krb5-libs-1.15.1-37.el7_7.2.x86_64....: [Errno 14] curl#7 - "Failed to connect to 2605:4d00::61: Network is unreachable"
Trying other mirror.
--------------------------------------------------------------------------------
Total 2.8 MB/s | 46 MB 00:16
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : libcom_err-1.42.9-16.el7.x86_64 1/65
Updating : krb5-libs-1.15.1-37.el7_7.2.x86_64 2/65
Updating : 1:openssl-libs-1.0.2k-19.el7.x86_64 3/65
Installing : apr-1.4.8-5.el7.x86_64 4/65
Installing : apr-util-1.5.2-6.el7.x86_64 5/65
Updating : 1:openssl-1.0.2k-19.el7.x86_64 6/65
Updating : 32:bind-license-9.11.4-9.P2.el7.noarch 7/65
Updating : 32:bind-libs-lite-9.11.4-9.P2.el7.x86_64 8/65
Updating : 12:dhcp-libs-4.2.5-77.el7.centos.x86_64 9/65
Installing : avahi-libs-0.6.31-19.el7.x86_64 10/65
Installing : 1:cups-libs-1.6.3-40.el7.x86_64 11/65
Installing : 1:cups-client-1.6.3-40.el7.x86_64 12/65
Updating : 12:dhcp-common-4.2.5-77.el7.centos.x86_64 13/65
Installing : 32:bind-libs-9.11.4-9.P2.el7.x86_64 14/65
Installing : 32:bind-utils-9.11.4-9.P2.el7.x86_64 15/65
Installing : httpd-tools-2.4.6-90.el7.centos.x86_64 16/65
Installing : 32:bind-export-libs-9.11.4-9.P2.el7.x86_64 17/65
Installing : MySQL-python-1.2.5-1.el7.x86_64 18/65
Installing : cyrus-sasl-plain-2.1.26-23.el7.x86_64 19/65
Installing : cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 20/65
Installing : libkadm5-1.15.1-37.el7_7.2.x86_64 21/65
Installing : mailx-12.5-19.el7.x86_64 22/65
Installing : libcom_err-devel-1.42.9-16.el7.x86_64 23/65
Updating : e2fsprogs-libs-1.42.9-16.el7.x86_64 24/65
Updating : libss-1.42.9-16.el7.x86_64 25/65
Installing : ed-1.9-4.el7.x86_64 26/65
Installing : spax-1.5.2-13.el7.x86_64 27/65
Installing : time-1.7-45.el7.x86_64 28/65
Installing : pcre-devel-8.32-17.el7.x86_64 29/65
Installing : python-psycopg2-2.5.1-4.el7.x86_64 30/65
Installing : libxslt-1.1.28-5.el7.x86_64 31/65
Installing : at-3.1.13-24.el7.x86_64 32/65
Installing : patch-2.7.1-12.el7_7.x86_64 33/65
Installing : bc-1.06.95-13.el7.x86_64 34/65
Installing : libsepol-devel-2.5-10.el7.x86_64 35/65
Installing : libselinux-devel-2.5-14.1.el7.x86_64 36/65
Installing : zlib-devel-1.2.7-18.el7.x86_64 37/65
Installing : libverto-devel-0.2.5-4.el7.x86_64 38/65
Installing : centos-logos-70.0.6-3.el7.centos.noarch 39/65
Installing : redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64 40/65
Installing : m4-1.4.16-10.el7.x86_64 41/65
Installing : redhat-lsb-core-4.1-27.el7.centos.1.x86_64 42/65
Installing : mailcap-2.1.41-2.el7.noarch 43/65
Installing : httpd-2.4.6-90.el7.centos.x86_64 44/65
Installing : 1:mod_ssl-2.4.6-90.el7.centos.x86_64 45/65
Installing : fuse-2.9.2-11.el7.x86_64 46/65
Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 47/65
Installing : krb5-devel-1.15.1-37.el7_7.2.x86_64 48/65
Installing : 1:openssl-devel-1.0.2k-19.el7.x86_64 49/65
Installing : fuse-libs-2.9.2-11.el7.x86_64 50/65
Installing : cloudera-manager-agent-6.3.1-1466458.el7.x86_64 51/65
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
Updating : e2fsprogs-1.42.9-16.el7.x86_64 52/65
Updating : 12:dhclient-4.2.5-77.el7.centos.x86_64 53/65
Cleanup : 12:dhclient-4.2.5-68.el7.centos.1.x86_64 54/65
Cleanup : 32:bind-libs-lite-9.9.4-72.el7.x86_64 55/65
Cleanup : 1:openssl-1.0.2k-16.el7.x86_64 56/65
Cleanup : e2fsprogs-1.42.9-13.el7.x86_64 57/65
Cleanup : 12:dhcp-common-4.2.5-68.el7.centos.1.x86_64 58/65
Cleanup : 12:dhcp-libs-4.2.5-68.el7.centos.1.x86_64 59/65
Cleanup : 32:bind-license-9.9.4-72.el7.noarch 60/65
Cleanup : krb5-libs-1.15.1-34.el7.x86_64 61/65
Cleanup : 1:openssl-libs-1.0.2k-16.el7.x86_64 62/65
Cleanup : e2fsprogs-libs-1.42.9-13.el7.x86_64 63/65
Cleanup : libss-1.42.9-13.el7.x86_64 64/65
Cleanup : libcom_err-1.42.9-13.el7.x86_64 65/65
Verifying : fuse-libs-2.9.2-11.el7.x86_64 1/65
Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 2/65
Verifying : fuse-2.9.2-11.el7.x86_64 3/65
Verifying : mailcap-2.1.41-2.el7.noarch 4/65
Verifying : httpd-tools-2.4.6-90.el7.centos.x86_64 5/65
Verifying : libcom_err-devel-1.42.9-16.el7.x86_64 6/65
Verifying : httpd-2.4.6-90.el7.centos.x86_64 7/65
Verifying : apr-1.4.8-5.el7.x86_64 8/65
Verifying : cloudera-manager-agent-6.3.1-1466458.el7.x86_64 9/65
Verifying : e2fsprogs-libs-1.42.9-16.el7.x86_64 10/65
Verifying : m4-1.4.16-10.el7.x86_64 11/65
Verifying : redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64 12/65
Verifying : 1:cups-client-1.6.3-40.el7.x86_64 13/65
Verifying : centos-logos-70.0.6-3.el7.centos.noarch 14/65
Verifying : 1:cups-libs-1.6.3-40.el7.x86_64 15/65
Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 16/65
Verifying : krb5-devel-1.15.1-37.el7_7.2.x86_64 17/65
Verifying : libverto-devel-0.2.5-4.el7.x86_64 18/65
Verifying : zlib-devel-1.2.7-18.el7.x86_64 19/65
Verifying : libsepol-devel-2.5-10.el7.x86_64 20/65
Verifying : libselinux-devel-2.5-14.1.el7.x86_64 21/65
Verifying : cyrus-sasl-plain-2.1.26-23.el7.x86_64 22/65
Verifying : bc-1.06.95-13.el7.x86_64 23/65
Verifying : e2fsprogs-1.42.9-16.el7.x86_64 24/65
Verifying : cyrus-sasl-gssapi-2.1.26-23.el7.x86_64 25/65
Verifying : patch-2.7.1-12.el7_7.x86_64 26/65
Verifying : krb5-libs-1.15.1-37.el7_7.2.x86_64 27/65
Verifying : 32:bind-export-libs-9.11.4-9.P2.el7.x86_64 28/65
Verifying : libss-1.42.9-16.el7.x86_64 29/65
Verifying : at-3.1.13-24.el7.x86_64 30/65
Verifying : libxslt-1.1.28-5.el7.x86_64 31/65
Verifying : apr-util-1.5.2-6.el7.x86_64 32/65
Verifying : python-psycopg2-2.5.1-4.el7.x86_64 33/65
Verifying : 1:mod_ssl-2.4.6-90.el7.centos.x86_64 34/65
Verifying : 1:openssl-devel-1.0.2k-19.el7.x86_64 35/65
Verifying : avahi-libs-0.6.31-19.el7.x86_64 36/65
Verifying : 12:dhcp-common-4.2.5-77.el7.centos.x86_64 37/65
Verifying : 32:bind-libs-lite-9.11.4-9.P2.el7.x86_64 38/65
Verifying : 32:bind-utils-9.11.4-9.P2.el7.x86_64 39/65
Verifying : MySQL-python-1.2.5-1.el7.x86_64 40/65
Verifying : 12:dhcp-libs-4.2.5-77.el7.centos.x86_64 41/65
Verifying : 1:openssl-1.0.2k-19.el7.x86_64 42/65
Verifying : pcre-devel-8.32-17.el7.x86_64 43/65
Verifying : libcom_err-1.42.9-16.el7.x86_64 44/65
Verifying : time-1.7-45.el7.x86_64 45/65
Verifying : spax-1.5.2-13.el7.x86_64 46/65
Verifying : libkadm5-1.15.1-37.el7_7.2.x86_64 47/65
Verifying : 32:bind-license-9.11.4-9.P2.el7.noarch 48/65
Verifying : ed-1.9-4.el7.x86_64 49/65
Verifying : 32:bind-libs-9.11.4-9.P2.el7.x86_64 50/65
Verifying : redhat-lsb-core-4.1-27.el7.centos.1.x86_64 51/65
Verifying : mailx-12.5-19.el7.x86_64 52/65
Verifying : 12:dhclient-4.2.5-77.el7.centos.x86_64 53/65
Verifying : 12:dhclient-4.2.5-68.el7.centos.1.x86_64 54/65
Verifying : 1:openssl-libs-1.0.2k-16.el7.x86_64 55/65
Verifying : libss-1.42.9-13.el7.x86_64 56/65
Verifying : 1:openssl-1.0.2k-16.el7.x86_64 57/65
Verifying : 32:bind-libs-lite-9.9.4-72.el7.x86_64 58/65
Verifying : e2fsprogs-libs-1.42.9-13.el7.x86_64 59/65
Verifying : 12:dhcp-libs-4.2.5-68.el7.centos.1.x86_64 60/65
Verifying : 12:dhcp-common-4.2.5-68.el7.centos.1.x86_64 61/65
Verifying : libcom_err-1.42.9-13.el7.x86_64 62/65
Verifying : 32:bind-license-9.9.4-72.el7.noarch 63/65
Verifying : krb5-libs-1.15.1-34.el7.x86_64 64/65
Verifying : e2fsprogs-1.42.9-13.el7.x86_64 65/65

Installed:
cloudera-manager-agent.x86_64 0:6.3.1-1466458.el7

Dependency Installed:
MySQL-python.x86_64 0:1.2.5-1.el7
apr.x86_64 0:1.4.8-5.el7
apr-util.x86_64 0:1.5.2-6.el7
at.x86_64 0:3.1.13-24.el7
avahi-libs.x86_64 0:0.6.31-19.el7
bc.x86_64 0:1.06.95-13.el7
bind-export-libs.x86_64 32:9.11.4-9.P2.el7
bind-libs.x86_64 32:9.11.4-9.P2.el7
bind-utils.x86_64 32:9.11.4-9.P2.el7
centos-logos.noarch 0:70.0.6-3.el7.centos
cups-client.x86_64 1:1.6.3-40.el7
cups-libs.x86_64 1:1.6.3-40.el7
cyrus-sasl-gssapi.x86_64 0:2.1.26-23.el7
cyrus-sasl-plain.x86_64 0:2.1.26-23.el7
ed.x86_64 0:1.9-4.el7
fuse.x86_64 0:2.9.2-11.el7
fuse-libs.x86_64 0:2.9.2-11.el7
httpd.x86_64 0:2.4.6-90.el7.centos
httpd-tools.x86_64 0:2.4.6-90.el7.centos
keyutils-libs-devel.x86_64 0:1.5.8-3.el7
krb5-devel.x86_64 0:1.15.1-37.el7_7.2
libcom_err-devel.x86_64 0:1.42.9-16.el7
libkadm5.x86_64 0:1.15.1-37.el7_7.2
libselinux-devel.x86_64 0:2.5-14.1.el7
libsepol-devel.x86_64 0:2.5-10.el7
libverto-devel.x86_64 0:0.2.5-4.el7
libxslt.x86_64 0:1.1.28-5.el7
m4.x86_64 0:1.4.16-10.el7
mailcap.noarch 0:2.1.41-2.el7
mailx.x86_64 0:12.5-19.el7
mod_ssl.x86_64 1:2.4.6-90.el7.centos
openssl-devel.x86_64 1:1.0.2k-19.el7
patch.x86_64 0:2.7.1-12.el7_7
pcre-devel.x86_64 0:8.32-17.el7
python-psycopg2.x86_64 0:2.5.1-4.el7
redhat-lsb-core.x86_64 0:4.1-27.el7.centos.1
redhat-lsb-submod-security.x86_64 0:4.1-27.el7.centos.1
spax.x86_64 0:1.5.2-13.el7
time.x86_64 0:1.7-45.el7
zlib-devel.x86_64 0:1.2.7-18.el7

Dependency Updated:
bind-libs-lite.x86_64 32:9.11.4-9.P2.el7
bind-license.noarch 32:9.11.4-9.P2.el7
dhclient.x86_64 12:4.2.5-77.el7.centos
dhcp-common.x86_64 12:4.2.5-77.el7.centos
dhcp-libs.x86_64 12:4.2.5-77.el7.centos
e2fsprogs.x86_64 0:1.42.9-16.el7
e2fsprogs-libs.x86_64 0:1.42.9-16.el7
krb5-libs.x86_64 0:1.15.1-37.el7_7.2
libcom_err.x86_64 0:1.42.9-16.el7
libss.x86_64 0:1.42.9-16.el7
openssl.x86_64 1:1.0.2k-19.el7
openssl-libs.x86_64 1:1.0.2k-19.el7

Complete!
END (0)
remote package cloudera-manager-agent installed
Installing cloudera-manager-daemons package...
BEGIN sudo yum --disablerepo=* --enablerepo=cloudera* list installed cloudera-manager-daemons
Loaded plugins: fastestmirror
Installed Packages
cloudera-manager-daemons.x86_64 6.3.1-1466458.el7 @Cloudera-manager
END (0)
BEGIN echo oracle-j2sdk1.8 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons
oracle-j2sdk1.8 cloudera-manager-agent cloudera-manager-daemons
END (0)
BEGIN sudo yum --disablerepo=* --enablerepo=cloudera* info cloudera-manager-daemons
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Installed Packages
Name : cloudera-manager-daemons
Arch : x86_64
Version : 6.3.1
Release : 1466458.el7
Size : 1.3 G
Repo : installed
From repo : cloudera-manager
Summary : Provides daemons for monitoring Hadoop and related tools.
URL : http://www.cloudera.com
License : Proprietary
Description : This package includes daemons for monitoring and managing Hadoop.

END (0)
Version : 6.3.1
Release : 1466458.el7
BEGIN sudo yum -y install cloudera-manager-daemons
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.web-ster.com
* extras: mirror.web-ster.com
* updates: mirror.web-ster.com
Package cloudera-manager-daemons-6.3.1-1466458.el7.x86_64 already installed and latest version
Nothing to do
END (0)
remote package cloudera-manager-daemons installed
Installing Unlimited Strength Encryption policy files.
BEGIN rpm -ql oracle-j2sdk1.8 | grep "/usr/java/jdk1.8" | sort | head -n 1
/usr/java/jdk1.8.0_181-cloudera
END (0)
Java 8 prefix is /usr/java/jdk1.8.0_181-cloudera
Did not find expected /usr/java/jdk1.8.0_181-cloudera/jre/lib/security/US_export_policy.jar. Not copying new policy file.
Did not find expected /usr/java/jdk1.8.0_181-cloudera/jre/lib/security/local_policy.jar. Not copying new policy file.
Configuring Cloudera Manager Agent...
BEGIN grep server_host=##.##.##.## /etc/cloudera-scm-agent/config.ini
END (1)
BEGIN sudo cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.ini.orig
END (0)
BEGIN sudo sed -e 's/\(server_host=\).*/\1##.##.##.##/' -i /etc/cloudera-scm-agent/config.ini
END (0)
scm agent configured
Starting Cloudera Manager Agent...
BEGIN sudo systemctl status cloudera-scm-agent
* cloudera-scm-agent.service - Cloudera Manager Agent Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
Active: inactive (dead)
END (3)
BEGIN sudo systemctl start cloudera-scm-agent
END (0)
agent logs:
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/'
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.out' for reading: No such file or directory
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.out' for reading: No such file or directory
END (0)
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/'
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory
tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory
END (0)
end of agent logs.
scm agent started
Installation script completed successfully.
all done
closing logging file descriptor

And this is from /var/log/cloudera-scm-agent.log (note when I first tried to view I got permission denied until I chmod-ed it):

Spoiler

[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO ================================================================================
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO SCM Agent Version: 6.3.1
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Agent Protocol Version: 4
[16/Jan/2020 22:57:14 +0000] 14874 MainThread __init__ INFO Generating Agent UUID: '/var/lib/cloudera-scm-agent/uuid'
[16/Jan/2020 22:57:14 +0000] 14874 MainThread __init__ INFO Agent UUID file was last modified at 2020-01-16 22:57:14.822205
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Using Host ID: 8d43e9dd-426c-4b36-b640-dfdcc5f484d1
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Using directory: /run/cloudera-scm-agent
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Using supervisor binary path: /opt/cloudera/cm-agent/bin/../bin/supervisord
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Agent Logging Level: INFO
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Agent config:
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Security.use_tls = 0
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Security.max_cert_depth = 9
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.orphan_process_dir_staleness_threshold = 5184000
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.dns_resolution_collection_timeout_seconds = 30
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.orphan_process_dir_refresh_interval = 3600
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.local_filesystem_whitelist = ext2,ext3,ext4,xfs
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.stacks_log_bundle_max_bytes = 1073741824
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.metrics_url_timeout_seconds = 30.0
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.max_collection_wait_seconds = 10.0
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.task_metrics_timeout_seconds = 5.0
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.stacks_log_max_uncompressed_file_size_bytes = 5242880
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.scm_debug = INFO
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.server_host = ##.##.##.##
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.monitored_nodev_filesystem_types = nfs,nfs4,tmpfs
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.server_port = 7182
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.dns_resolution_collection_interval_seconds = 60
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO General.impala_profile_bundle_max_bytes = 1073741824
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Agent starting as pid 14874 user root(0) group root(0).
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found existing subsystem cpuacct at /sys/fs/cgroup/cpu,cpuacct
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found existing subsystem cpu at /sys/fs/cgroup/cpu,cpuacct
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found existing subsystem blkio at /sys/fs/cgroup/blkio
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found existing subsystem memory at /sys/fs/cgroup/memory
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found existing subsystem devices at /sys/fs/cgroup/devices
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found cgroups subsystem: cpu
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Adding more caps: {'writable_cgroup_dot_procs': True, 'default_cpu_shares': 1024, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True}
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found cgroups subsystem: cpuacct
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Adding more caps: {'has_cpuacct': True}
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found cgroups subsystem: memory
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Adding more caps: {'has_memory': True, 'default_memory_limit_in_bytes': 8796093022207, 'default_memory_soft_limit_in_bytes': 8796093022207}
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found cgroups subsystem: devices
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Adding more caps: {'has_devices': True}
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Found cgroups subsystem: blkio
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Adding more caps: {'has_blkio': True, 'default_blkio_weight': 1000}
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Ignoring cgroup avro_cap: default_memory_limit_in_bytes
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Ignoring cgroup avro_cap: default_blkio_weight
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Ignoring cgroup avro_cap: writable_cgroup_dot_procs
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Ignoring cgroup avro_cap: default_cpu_rt_runtime_us
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Ignoring cgroup avro_cap: default_memory_soft_limit_in_bytes
[16/Jan/2020 22:57:14 +0000] 14874 MainThread cgroups INFO Ignoring cgroup avro_cap: default_cpu_shares
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': 8796093022207, 'default_blkio_weight': 1000, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_memory_soft_limit_in_bytes': 8796093022207, 'has_cpuacct': True, 'default_cpu_shares': 1024, 'has_devices': True, 'has_blkio': True}
[16/Jan/2020 22:57:14 +0000] 14874 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[16/Jan/2020 22:57:14 +0000] 14874 MainThread filesystem_map INFO Using timeout of 2.000000
[16/Jan/2020 22:57:14 +0000] 14874 MainThread filesystem_map INFO Using join timeout of 0.100000
[16/Jan/2020 22:57:14 +0000] 14874 MainThread filesystem_map INFO Using tolerance of 60.000000
[16/Jan/2020 22:57:14 +0000] 14874 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4', 'xfs']
[16/Jan/2020 22:57:14 +0000] 14874 MainThread kt_renewer INFO Agent wide credential cache set to /var/run/cloudera-scm-agent/krb5cc_cm_agent_0
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Setting up supervisord event monitor.
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
[16/Jan/2020 22:57:14 +0000] 14874 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
[16/Jan/2020 22:57:14 +0000] 14874 MainThread metrics INFO Importing tasktracker metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/tasktracker/schema.json
[16/Jan/2020 22:57:14 +0000] 14874 MainThread ntp_monitor INFO Using timeout of 2.000000
[16/Jan/2020 22:57:15 +0000] 14874 MainThread dns_names INFO Using timeout of 30.000000
[16/Jan/2020 22:57:15 +0000] 14874 MainThread __init__ INFO Created DNS monitor.
[16/Jan/2020 22:57:15 +0000] 14874 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
[16/Jan/2020 22:57:15 +0000] 14874 MainThread daemon INFO Importing metric schema from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/schema.json
[16/Jan/2020 22:57:15 +0000] 14874 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CMF_PACKAGE_DIR': '/opt/cloudera/cm-agent/service', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_PARQUET_HOME': '/usr/lib/parquet', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/opt/cloudera/cm', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_SERVER_ROOT': '/opt/cloudera/cm', 'CDH_KUDU_HOME': '/usr/lib/kudu', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_HUE_HOME': '/usr/lib/hue', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[16/Jan/2020 22:57:15 +0000] 14874 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[16/Jan/2020 22:57:17 +0000] 14874 MainThread supervisor INFO Trying to connect to supervisor (Attempt 1)
[16/Jan/2020 22:57:17 +0000] 14874 MainThread supervisor INFO Supervisor version: 3.0, pid: 14873
[16/Jan/2020 22:57:17 +0000] 14874 MainThread supervisor INFO Successfully connected to supervisor
[16/Jan/2020 22:57:17 +0000] 14874 MainThread agent INFO Supervisor version: 3.0, pid: 14873
[16/Jan/2020 22:57:17 +0000] 14874 MainThread agent INFO Connecting to previous supervisor: agent-14873-1579215434.
[16/Jan/2020 22:57:19 +0000] 14874 MainThread supervisor INFO Triggering supervisord update.
[16/Jan/2020 22:57:19 +0000] 14874 MainThread _cplogging INFO [16/Jan/2020:22:57:19] ENGINE Bus STARTING
[16/Jan/2020 22:57:19 +0000] 14874 MainThread _cplogging INFO [16/Jan/2020:22:57:19] ENGINE Started monitor thread '_TimeoutMonitor'.
[16/Jan/2020 22:57:19 +0000] 14874 MainThread _cplogging INFO [16/Jan/2020:22:57:19] ENGINE Serving on http://127.0.0.1:9001
[16/Jan/2020 22:57:19 +0000] 14874 MainThread _cplogging INFO [16/Jan/2020:22:57:19] ENGINE Bus STARTED
[16/Jan/2020 22:57:19 +0000] 14874 MainThread daemon INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x7f87bb77c6d0>,)
[16/Jan/2020 22:57:19 +0000] 14874 MonitorDaemon-Scheduler daemon INFO Monitor ready to report: ('HostMonitor',)
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Setting default socket timeout to 45
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Failed to read available parcel file: [Errno 2] No such file or directory: '/var/lib/cloudera-scm-agent/active_parcels.json'
[16/Jan/2020 22:57:19 +0000] 14874 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.44 min:0.44 mean:0.44 max:0.44 LIFE_MAX:0.44
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO CM server guid: 5ffc5cb8-dfa8-40de-b50a-26bf7e9927a0
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Created /opt/cloudera/parcels
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Chowning /opt/cloudera/parcels to root (0) root (0)
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Chmod'ing /opt/cloudera/parcels to 0755
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Created /opt/cloudera/parcel-cache
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Chowning /opt/cloudera/parcel-cache to root (0) root (0)
[16/Jan/2020 22:57:19 +0000] 14874 MainThread agent INFO Chmod'ing /opt/cloudera/parcel-cache to 0755
[16/Jan/2020 22:57:19 +0000] 14874 MainThread parcel INFO Agent does create users/groups and apply file permissions
[16/Jan/2020 22:57:20 +0000] 14874 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[16/Jan/2020 22:57:20 +0000] 14874 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[16/Jan/2020 22:57:20 +0000] 14874 MainThread throttling_logger WARNING Failed parsing alternatives line: libnssckbi.so.x86_64 string index out of range link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
[16/Jan/2020 22:57:20 +0000] 14874 MainThread agent INFO Flood daemon (re)start attempt
[16/Jan/2020 22:57:21 +0000] 14874 MainThread agent INFO Created /opt/cloudera/parcels/.flood
[16/Jan/2020 22:57:21 +0000] 14874 MainThread agent INFO Chowning /opt/cloudera/parcels/.flood to cloudera-scm (997) cloudera-scm (994)
[16/Jan/2020 22:57:21 +0000] 14874 MainThread agent INFO Chmod'ing /opt/cloudera/parcels/.flood to 0755
[16/Jan/2020 22:57:22 +0000] 14874 MainThread agent INFO Reloading status server config, sending SIGHUP
[16/Jan/2020 22:57:22 +0000] 14874 MainThread agent INFO Active parcel list updated; recalculating component info.
[16/Jan/2020 22:57:22 +0000] 14874 MainThread throttling_logger INFO Identified java component java8 with full version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) for requested version .
[16/Jan/2020 23:07:23 +0000] 14874 MainThread heartbeat_tracker INFO HB stats (seconds): num:41 LIFE_MIN:0.01 min:0.01 mean:0.01 max:0.04 LIFE_MAX:0.44

Thanks you all again for taking a look.

Re: CM Install on EC2 Failing at Agent Heartbeat Check

New Contributor

More Info:

 

I attempted the identical workflow on an EC2 instance in a public VPC with a public DNS name and I do have have the same problem.  For my use I would much prefer an instance in our private VPC, however.  Does anyone have experience with Cloudera Manager Express on EC2 in a private subnet with a public IPv4 but no public DNS?

Re: CM Install on EC2 Failing at Agent Heartbeat Check

New Contributor

Further info:

 

Following the same install procedure on a public VPC based EC2 instance with a public DNS in addition to a public IPv4 does not create a similar issue.

 

At this point I think we are going to abandon CM as it is not really needed for our use case and we very much want to stay within our private VPC.

Don't have an account?
Coming from Hortonworks? Activate your account here