CDH Upgrade stalls at 50% distributed

[#1]

I'm trying to upgrade CDH from 5.9.0 to 5.9.13 (which is what {latest_supported} returns) but I'm running into problems.   I was able to use the parcels window to download the parcel for 5.9.3-1.cdh5.9.3.p0.4 no problem.   When I click distribute, it quickly gets to 50%, but then stalls.   Clicking on "Details" shows that it has completed 6/6 Distribution steps.   

Looking at /var/log/cloudera-scm-server/cloudera-scm-server.log gives very little help:

2018-08-10 09:16:49,997 INFO 417074583@agentServer-745696:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=37ms, min=0ms, max=342ms.
2018-08-10 09:16:49,997 INFO 417074583@agentServer-745696:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=16ms.
2018-08-10 09:17:50,054 INFO 983417458@agentServer-745700:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=37ms, min=0ms, max=342ms.
2018-08-10 09:17:50,054 INFO 983417458@agentServer-745700:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=16ms.
2018-08-10 09:18:50,093 INFO 954942595@agentServer-745687:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=37ms, min=0ms, max=342ms.
2018-08-10 09:18:50,093 INFO 954942595@agentServer-745687:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=16ms.
2018-08-10 09:18:56,513 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully.
2018-08-10 09:19:50,166 INFO 235172469@agentServer-745703:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=38ms, min=0ms, max=342ms.
2018-08-10 09:19:50,166 INFO 235172469@agentServer-745703:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=16ms.
2018-08-10 09:20:50,225 INFO 1657097256@agentServer-745702:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=38ms, min=0ms, max=342ms.
2018-08-10 09:20:50,225 INFO 1657097256@agentServer-745702:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=16ms.
2018-08-10 09:21:32,740 INFO 1273326417@scm-web-758047:com.cloudera.parcel.components.ParcelManagerImpl: Distributing parcel CDH:5.9.3-1.cdh5.9.3.p0.4 on cluster cluster
2018-08-10 09:22:02,895 INFO 235172469@agentServer-745703:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=40ms, min=0ms, max=343ms.
2018-08-10 09:22:02,895 INFO 235172469@agentServer-745703:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=16ms.

6 node cluster running centos 6.6, if that helps.

I appreciate any guidance to help me get this upgrade moving.   Ultimately I need to get CDH & CM to 5.15, but I wanted to have a success under my belt before I tackled that.

[#2 · Accepted solution]

Restarted my cloudera manager server, problem magically went away.

[#3]

By which I mean I literally rebooted the box.

[#4]

@supahcraig,

I am glad you were able to resolve the problem.

Usually if the upgrade stalls during distribution you can view details about which hosts have successfully downloaded the parcel via the the parcels page.  Clicking the progress bar will open a page that shows the progress for each host.

Once you find a host or hosts that have errors or are not completing, you can mouse over or click on the bar to view more details including the hostname.

Once you have the hostname, you can ssh to that host and review the agent log there (/var/log/cloudera-scm-agent/cloudera-scm-agent.log)

The agent actually downloads the parcel file so you can see information about any problems in the agent log.

Here is a page that shows more about the parcels:

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_ig_parcels.html

Cheers!