Status: Failed to receive heartbeat from agent

[#1]

I am facing issue ERROR: Toplevel exception SPAWN status error.

Hi,

I'm trying to set up the Cloudera data platform private cloud base in my VirtualBox with ubuntu 20.04 for trial purposes.

Virtualbox specs:

4GB RAM

80GB Disk

Ubuntu partition: /opt - 30GB   /var - 30G      /usr - 7GB      / 17GB

as per Cloudera manager server specifications

I have installed the Cloudera Manager 7.4.4

and uploaded the Cloudera license to install the latest CDH 7.1.7 for POC purposes.

hostname: master

/etc/hosts file:

127.0.0.1 localhost 

10.0.2.15 master

NOTE: While specifying the hosts in cloudera manager. I choose localhost to connect. (scanned) wasn't able to go with master. I am installing these steps in ubuntu virtualbox.

Installation script completed successfully.

all done
closing logging file descriptor
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Adding more caps: {'has_memory':
True, 'default_memory_limit_in_bytes': 8796093022207, 'default_memory_soft_limit_in_bytes':
8796093022207}
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Found cgroups subsystem: devices
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Adding more caps: {'has_devices':
True}
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Ignoring cgroup avro_cap:
default_memory_limit_in_bytes
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Ignoring cgroup avro_cap:
default_blkio_weight
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Ignoring cgroup avro_cap:
writable_cgroup_dot_procs
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Ignoring cgroup avro_cap:
default_cpu_rt_runtime_us
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Ignoring cgroup avro_cap:
default_memory_soft_limit_in_bytes
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread cgroups INFO Ignoring cgroup avro_cap:
default_cpu_shares
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread agent INFO Found cgroups capabilities:
{'has_memory': True, 'default_memory_limit_in_bytes': 8796093022207, 'default_blkio_weight': -1,
'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': -1, 'has_cpu': True,
'default_memory_soft_limit_in_bytes': 8796093022207, 'has_cpuacct': True, 'default_cpu_shares': 1024,
'has_devices': True, 'has_blkio': True}
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread filesystem_map INFO Monitored nodev filesystem
types: ['nfs', 'nfs4', 'tmpfs']
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread filesystem_map INFO Using timeout of 2.000000
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread filesystem_map INFO Using join timeout of
0.100000
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread filesystem_map INFO Using tolerance of 60.000000
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread filesystem_map INFO Local filesystem types
whitelist: ['ext2', 'ext3', 'ext4', 'xfs']
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread kt_renewer INFO Agent wide credential cache set to
/var/run/cloudera-scm-agent/krb5cc_cm_agent_0
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread agent INFO Setting up supervisord event monitor.
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread agent INFO Using metrics_url_timeout_seconds of
30.000000
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread agent INFO Using task_metrics_timeout_seconds of
5.000000
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread agent INFO Using max_collection_wait_seconds of
10.000000
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread metrics INFO Importing tasktracker metric schema
from file /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/tasktracker/schema.json
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread ntp_monitor INFO Using timeout of 2.000000
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread dns_names INFO Using timeout of 30.000000
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread __init__ INFO Created DNS monitor.
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread stacks_collection_manager INFO Using
max_uncompressed_file_size_bytes: 5242880
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread daemon INFO Importing metric schema from file
/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/schema.json
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread agent INFO Supervised processes will add the
following to their environment (in addition to the supervisor's env): {'CMF_PACKAGE_DIR':
'/opt/cloudera/cm-agent/service', 'CM_STATUS_CODES': u'STATUS_NONE
HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED

JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE',
'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CDH_SQOOP2_HOME':
'/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'JSVC_HOME':
'/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
'CLOUDERA_POSTGRESQL_JDBC_JAR': '/opt/cloudera/cm/lib/postgresql-42.2.14.jre7.jar',
'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar',
'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',

'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HBASE_CONNECTORS_HOME': '/usr/lib/hbase-
connectors', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka',

'CDH_KNOX_HOME': '/usr/lib/knox', 'CDH_PARQUET_HOME': '/usr/lib/parquet',
'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/opt/cloudera/cm', 'CDH_IMPALA_HOME':

'/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-
hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin',

'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_SERVER_ROOT': '/opt/cloudera/cm',
'CDH_KUDU_HOME': '/usr/lib/kudu', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'ORACLE_HOME': '/usr/share/oracle/instantclient',
'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_HBASE_FILESYSTEM_HOME':
'/usr/lib/hbase-filesystem', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME':
'/usr/lib/hbase', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml',
'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_HUE_HOME': '/usr/lib/hue',
'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-
mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_TEZ_HOME':
'/usr/lib/tez', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HIVE_HOME': '/usr/lib/hive',
'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
>>[29/Jul/2022 02:18:15 +0530] 19336 MainThread agent INFO To override these variables, use
/etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is
installed from parcels.
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread supervisor INFO Trying to connect to supervisor
(Attempt 1)
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread supervisor INFO Supervisor version: 3.4.0, pid:
11512
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread supervisor INFO Successfully connected to
supervisor
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread agent INFO Supervisor version: 3.4.0, pid: 11512
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread agent INFO Connecting to previous supervisor:
agent-11512-1659038861.
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread supervisor INFO Triggering supervisord update.
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread _cplogging INFO [29/Jul/2022:02:18:17] ENGINE
Bus STARTING
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread _cplogging INFO [29/Jul/2022:02:18:17] ENGINE
Started monitor thread '_TimeoutMonitor'.
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread _cplogging INFO [29/Jul/2022:02:18:17] ENGINE
Serving on http://127.0.0.1:9001
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread _cplogging INFO [29/Jul/2022:02:18:17] ENGINE
Bus STARTED
>>[29/Jul/2022 02:18:17 +0530] 19336 MainThread status_server INFO Status server url is
http://master:9000/
>>[29/Jul/2022 02:18:21 +0530] 19336 MainThread main ERROR Top-level exception: <Fault 50:
'SPAWN_ERROR: status_server'>
>>Traceback (most recent call last):
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/main.py", line 109, in main_impl
>> ag.start(legacy_supervisor)
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 898, in start

>> self.supervisor_client.start_process(STATUS_SERVER_PROC)
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/__init__.py", line 531, in new_fn
>> return fn(self, *args, **kwargs)
>> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/supervisor.py", line 378, in start_process
>> raise RetryableProcessException(fault)
>>RetryableProcessException: <Fault 50: 'SPAWN_ERROR: status_server'>

please suggest me ways to solve this error.

[#2]

Did you find solution to this ?

[#3]

Hello,

Thank you for reaching out

Can you please share the error screenshot?

Do you see the host on CM UI?

Can you compare the /etc/cloudera-scm-agent/config.ini from the working node it that correct?

Also share the output of following command

# /opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf status