Could not upgrade hdp from 2.3.4 to 2.4.2 with Ambari errors

[#1]

We were running hdp version 2.3.4 and attempted to upgrade to version 2.4.2. During the package install process, which pushes the install packages to all nodes in our cluster, we ran into an issue where the install packages could not be loaded to our edge nodes. The errors displayed by Ambari were:

All hosts should have target version installed Reason: The following hosts must have version 2.4.2.0-258 installed: server1.domain.com and server2.domain.com. Failed on: server1.domain.com, server2.domain.com.

Install packages must be re-run Reason: Hosts in cluster [HAL_PROD,HDP,2.4,2.4.2.0-258] are in INSTALL_FAILED state because Install Packages had failed. Please re-run Install Packages, if necessary place following hosts in Maintenance mode: server1.domain.com, server2.domain.com., Failed on: server1.domain.com, server2.domain.com.

How can I fix this issue?

[#2]

Nick -

Can you check quickly what type of disk space you have in the edge nodes? Also, did all the other Data Nodes complete the installation properly? So your problem is only on the edge nodes?

Alex -

[#3]

All other nodes installed successfully... the only problems were the edge nodes. I will check the space there. Thank you Alex for your quick reply!

[#4]

Nick -

Could you also paste the error that came during package install on Edge nodes? This can be seen from Ambari's "Background Operations" Window under "Install Version" task

Besides you could also do a "Re-Install" in "Manage Versions" page which would attempt a retry of package installation on the failed nodes. This is useful for scenarios where package install could have failed due to excessive load on network or the repos being temporarily unavailable

Thanks, Vivek

[#5]

Vivek - the error I pasted in my original post was what I received. It appears to be a disk space issue.

[#6]

Hey Nick - That error generally comes when you try to start the upgrade. It is printed as part of the pre-checks done for upgrade. I was looking for the error that would have come in an earlier step which is during the failed package installation.

Good to know that you found the disk space as the cause of the problem. In that case since you freed up the space, a reinstall of packages should resolve the issue

Let us know if you still see any issues

Thanks, Vivek

[#7]

We found some Ambari errors related to disk space - particularly the edge nodes. We increased the space there.

[#8 · Accepted solution]

Nick -

Re run the install (should be a drop down arrow next to the Upgrade button >> Reinstall);

Let me know how it goes.

Alex -

[#9]

Alex - that did it! Thanks for your help. Much appreciated.