CDP 7.2.4 upgrade - cloudera agents not starting

[#1]

Hello Experts,

I have upgraded CDH 5.16 to CDP 7.2.4 Cloudera Manager only( not the runtime yet). After upgrade the Cloudera Manager server is running good including agent. But the Cloudera Agents on the rest of the nodes wont start when running $ sudo systemctl restart cloudera-scm-agent

The agents are still at 5.16 as the final step for upgrade does show the upgrade button for the agents. Before the upgrade the agents were running fine. Also the agents where shut down during the Cloudera Manager upgrade. How to start the agents and upgrade them to CDP 7.2.4 now.

When trying to restart agents It just shows:

$ sudo systemctl status cloudera-scm-agent
● cloudera-scm-agent.service - LSB: Cloudera SCM Agent
Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)
Active: active (exited) since Tue 2021-01-12 17:52:10 EST; 7s ago
Docs: man:systemd-sysv-generator(8)
Process: 14093 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS)
Process: 14210 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS)

In the /var/log/cloudera-scm-agent/cloudera-scm-agent.log I see some below messages:

u'KS_INDEXER', u'ZOOKEEPER-SERVER', u'SERVER', u'HIVE_ON_TEZ', u'HIVE_ON_TEZ', u'HIVE_LLAP', u'HIVE_LLAP', u'KEYTRUSTEE_SERVER', u'KEYTRUSTEE_SERVER', u'SCHEMAREGISTRY-SCHEMA_REGISTRY_SERVER', u'SCHEMA_REGISTRY_SERVER', u'OZONE', u'OZONE', u'MAPREDUCE-JOBTRACKER', u'JOBTRACKER', u'THALES_KMS-HSMKP_THALES', u'HSMKP_THALES'], u'flood_seed_timeout': 100, u'eventserver_port': 7185}
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/agent.py", line 1566, in handle_heartbeat_response
self._handle_heartbeat_response(response)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/agent.py", line 1581, in _handle_heartbeat_response
self.java_home_config = self.extra_configs['JAVA_HOME']
KeyError: 'JAVA_HOME'
 

Any thoughts?

Thanks!

[#2 · Accepted solution]

This was resolved by manually upgrading the agents in the other nodes which were still at CM5.16 by running the commands below:

First update the /etc/yum.repos.d/cloudera-manager.repo on the nodes with the proper repo for CM7.2.4.
After that run command on each agent to be upgraded:
$yum upgrade cloudera-manager-daemons cloudera-manager-agent

After that restart the cloudera manager agents on all nodes:
$ systemctl restart cloudera-scm-agent

Next go to Cloudera Manager GUI and restart the Cloudera Management Service. After that restart the CDH Cluster all services. This should resolve the issues.