Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP OS Upgrade/Patching Best practices

avatar

Are there any best practices/ documentation around patching or upgrading an OS (e.g. upgrading CentOS 6 --> 7 or security patching ) while the cluster is running?

Thanks,

1 ACCEPTED SOLUTION

avatar

This post by Lester Martin sums up really well - https://martin.atlassian.net/wiki/pages/viewpage.action?pageId=36044812

Here is a summary -

  1. Since most OS patching / upgrade requires reboot, it is best to schedule such an activity around a scheduled outage.
  2. It is also recommended to go through the exercise in a lower level environment prior to applying changes in a PROD environment.
  3. In order to be apply the changes while the cluster is up, the patch/upgrade will have to be applied in rolling manner by first stopping the components in the host from Ambari, then applying changes, rebooting host and then starting the Hadoop services from Ambari. Repeat for each host.

This process will have to be scripted for a large size cluster. The steps of stopping and starting the cluster can be performed by uzing Ambari APIs.

View solution in original post

9 REPLIES 9

avatar

This post by Lester Martin sums up really well - https://martin.atlassian.net/wiki/pages/viewpage.action?pageId=36044812

Here is a summary -

  1. Since most OS patching / upgrade requires reboot, it is best to schedule such an activity around a scheduled outage.
  2. It is also recommended to go through the exercise in a lower level environment prior to applying changes in a PROD environment.
  3. In order to be apply the changes while the cluster is up, the patch/upgrade will have to be applied in rolling manner by first stopping the components in the host from Ambari, then applying changes, rebooting host and then starting the Hadoop services from Ambari. Repeat for each host.

This process will have to be scripted for a large size cluster. The steps of stopping and starting the cluster can be performed by uzing Ambari APIs.

avatar

Thanks @bsaini . Do you know how HDFS rebalancing would work during an OS upgrade/rebalance? When would HDFS start trying to rebalance the data residing on the DataNode being bounced?

avatar
Explorer

Thx @bsani

We have sat HA on our name nodes while we don't want the cluster to be unawailable.

So is there a best practice for doing patching on a cluster that is supposed to be available 24/7.

How to avoid rebalancing during patching

When upgrading datanodes in chuncks, is there a way to make sure that replica of a data block is available on one of the servers alive

/Claus

avatar
Master Mentor

avatar
Expert Contributor

Hi Neeraj.. Is there any latest hortonworks documentation to know step by step procedure to perform O.S upgrade for hadoop cluster?

avatar
Master Mentor

avatar
Expert Contributor

We enabled HA for Namenode and ResourceManager. What are the actions we need to take care while doing O.S updates on HDP cluster?

avatar
Master Mentor

go through each server one by one, put services in maintenance mode on that host and stop them. Upgrade OS to latest minor version, reboot if necessary and restart services. Go to the next. @Ram D

avatar
Expert Contributor

Thank you @Artem Ervits