Support Questions
Find answers, ask questions, and share your expertise

CM 5.5.3 --> 5.7.0 upgrade fails - No response on port 7180 (with solution but perhaps a bug)

Highlighted

CM 5.5.3 --> 5.7.0 upgrade fails - No response on port 7180 (with solution but perhaps a bug)

New Contributor

Hi guys, I have run into some problems upgrading CM to 5.7.0.

 

- Have: CDH 5.5.3 from Feb 2016. Full 1+3 node cluster, not a quick-thing. Cluster is internet connected and not a production environment.

- Followed the Cloudera 5 --> 5.7 upgrade, got to the parcel finder, found nothing,

(I.e. got to "Run the Upgrade Wizard" and (4) and there was no parcels even with added URLs)

http://www.cloudera.com/documentation/enterprise/latest/topics/install_upgrade_to_cdh57_parcels.html...

 

Decided to try and upgrade CM from 5(.5.3) to CM 5.7.0,

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ag_upgrade_cm5.html#concept_m35_t1...

 

- Followed the CM upgrade (package version, not tarball) and did all the tedious pre-checks and everything checked out. Ran the

yum upgrade cloudera-manager-server cloudera-manager-daemons cloudera-manager-server-db-2 cloudera-manager-agent

..and everything worked/installed to 5.7.0.

 

I ran both the stopping cmds for the CM before, ie both for embedded and for PostGreSQL and it seems to run Post

 

I got no response on 7180 after having started all 3 services,

(They were stopped the same way prior to the yum upgrades)

service cloudera-scm-server-db start

service cloudera-scm-server start

service cloudera-scm-agent start

 

ps -ef | grep -i cloudera-scm

..gav me a lot of running processes that still ran 5.5.3 despite having been shut down from CM.

 

Stopped all 3 CM services again and shot these ps listed processes down afterwards with kill -9.

 

Then restarted the CM services again and 7180 worked.

 

I and my colleague who did this with me are sure we did shut down all the CM processes from within CM as instructed in the upgrade instructions.

 

Wanted to post a solution in case others get the same and in case we didn't both have a mental lookover in the upgrade procedure and both missed the shutdowns earlier; i.e. what seems to be an unclean shut down of CM services and some manual help needed there. Also thanks to user alexx on this forum for a similar post that helped.

 

Edit: And the anonymized lines from ",ps -ef | grep -i cloudera-scm",

clouder+ 3508 16036 0 May02 ? 00:02:18 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-NAVIGATOR-XXXXXX.XXX.XXX.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dnavigator.schema.dir=/usr/share/cmf/cloudera-navigator-audit-server/schema -Dnavigator.auditModels.dir=/usr/share/cmf/cloudera-navigator-audit-server/auditModels -Xms627048448 -Xmx627048448 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/535-cloudera-mgmt-NAVIGATOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/plugins/event-publish-5.5.3-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.5.3.jar:/usr/share/cmf/cloudera-navigator-audit-server/* com.cloudera.navigator.NavigatorMain --conf-dir /run/cloudera-scm-agent/process/535-cloudera-mgmt-NAVIGATOR
clouder+ 3564 16036 0 May02 ? 00:01:50 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-ALERTPUBLISHER-XXXXXX.XXX.XXX.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xms268435456 -Xmx268435456 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/536-cloudera-mgmt-ALERTPUBLISHER:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.enterprise.alertpublisher.AlertPublisher
clouder+ 3616 16036 8 May02 ? 00:41:32 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-REPORTSMANAGER-XXXXXX.XXX.XXX.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dheadlamp.schema.dir=/usr/share/cmf/schema -Xms721420288 -Xmx721420288 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/537-cloudera-mgmt-REPORTSMANAGER:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.headlamp.HeadlampServer --db-conf-dir /run/cloudera-scm-agent/process/537-cloudera-mgmt-REPORTSMANAGER --db-conf-file headlamp.db.properties --mgmt-home /usr/share/cmf
clouder+ 3669 16036 0 May02 ? 00:02:48 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-ACTIVITYMONITOR-XXXXXX.XXX.XXX.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Xms721420288 -Xmx721420288 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/538-cloudera-mgmt-ACTIVITYMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.cmon.firehose.Main --pipeline-type ACTIVITY_MONITORING_TREE --mgmt-home /usr/share/cmf
clouder+ 3721 16036 2 May02 ? 00:10:03 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-XXXXXX.XXX.XXX.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xms627048448 -Xmx627048448 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/539-cloudera-mgmt-EVENTSERVER:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.cmf.eventcatcher.server.EventCatcherService
clouder+ 3773 16036 80 May02 ? 06:18:54 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-XXXXXX.XXX.XXX.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -XX:PermSize=128m -Dsun.rmi.transport.tcp.handshakeTimeout=10000 -Dsun.rmi.transport.tcp.responseTimeout=10000 -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/540-cloudera-mgmt-SERVICEMONITOR -Xms268435456 -Xmx268435456 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/540-cloudera-mgmt-SERVICEMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.cmon.firehose.Main --pipeline-type SERVICE_MONITORING --mgmt-home /usr/share/cmf
clouder+ 3838 16036 26 May02 ? 02:03:35 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-HOSTMONITOR-XXXXXX.XXX.XXX.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/usr/share/cmf/schema -Dlibrary.leveldbjni.path=/run/cloudera-scm-agent/process/541-cloudera-mgmt-HOSTMONITOR -Xms268435456 -Xmx268435456 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /run/cloudera-scm-agent/process/541-cloudera-mgmt-HOSTMONITOR:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/* com.cloudera.cmon.firehose.Main --pipeline-type HOST_MONITORING --mgmt-home /usr/share/cmf
clouder+ 3896 16036 0 May02 ? 00:04:18 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-NAVIGATORMETASERVER-XXXXXX.XXX.XXX.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xms1442840576 -Xmx1442840576 -XX:MaxPermSize=128m -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -cp /usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/cloudera-navigator-server/nav-server-2.4.3.jar:/usr/share/cmf/cloudera-navigator-server/jars/*:/usr/share/cmf/lib/plugins/event-publish-5.5.3-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.5.3.jar -Dlog4j.configuration=file:///run/cloudera-scm-agent/process/542-cloudera-mgmt-NAVIGATORMETASERVER/log4j.properties -Dnavigator.auditModels.dir=/usr/share/cmf/cloudera-navigator-audit-server/auditModels com.cloudera.nav.server.NavServer /run/cloudera-scm-agent/process/542-cloudera-mgmt-NAVIGATORMETASERVER/cloudera-navigator.properties /run/cloudera-scm-agent/process/542-cloudera-mgmt-NAVIGATORMETASERVER/cloudera-navigator-cm-auth.properties /run/cloudera-scm-agent/process/542-cloudera-mgmt-NAVIGATORMETASERVER/db.navms.properties
clouder+ 9787 1 0 Feb23 ? 00:09:23 /usr/bin/postgres -D /var/lib/cloudera-scm-server-db/data -k /var/run/cloudera-scm-server/
root 16054 16036 0 Feb24 ? 00:00:00 /usr/lib64/cmf/agent/build/env/bin/python /usr/lib64/cmf/agent/src/cmf/supervisor_listener.py -l /var/log/cloudera-scm-agent/cmf_listener.log /run/cloudera-scm-agent/events

 

Not sure why so much survived after being shut down, but perhaps some admin priviledges was not in order and blocked a clean shutdown of them.

1 REPLY 1
Highlighted

Re: CM 5.5.3 --> 5.7.0 upgrade fails - No response on port 7180 (with solution but perhaps a bug)

Super Guru

Hi raolesen,

 

Thank you for your detailed recap of your experience.

 

When you shut down Cloudera Management Service roles from Cloudera manager, it is up to the agent to heartbeat to Cloudera Manager to receive the shutdown instruction.

 

The agent then needs to signal the supervisor to shut down the process.  It is possible that something could have gone wrong during that shutdown.  You could check in the /var/log/cloudera-scm-agent/cloudera-scm-agent.log and possibly the supervisord.log for clues.

 

When using the embedded database, it will not shut down cleanly if any servers are using it.

 

There are a few things that could have happened, but it is tough to say without digging in the logs.

 

If you run into something odd like this in the future, we can help try to figure out what went wrong, too.

I'm gald that the kill helped in your case!

 

-Ben