How to increase timeout value for Namenode restarts when performing HDP Upgrade ?

[#1]

During upgrade, if Namenode restarts timeout, it may not appear to be a problem as the request times out from the Ambari UI but the restart process continue to run in the background.

However, this can lead to inconsistencies in Ambari database and cause further issues at Finalize upgrade step.

Note: This article is only useful upto Ambari-2.5.x version and must be performed before starting the upgrade process.

Ambari-2.6 onwards it is a one step change where you only need to update the ambari.properties file instead of all the xml changes listed below.

For Ambari-2.6 onwards: refer this document https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/preparing_to_upgr...

Caution: The steps described below are a hack! It is not recommended that we go about making changes to upgrade XML files based on our needs. Please exercise caution and measure your risks before following the steps.

You can increase the timeouts for namenode restart using the following steps before you start the upgrade process:

Step 1: Locate the upgrade file on Ambari server host. If we are upgrading from HDP-2.5 to HDP-2.6 then :

/var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml [Express Upgrade]

/var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml [Rolling Upgrade]

Step 2: Change

    <service name="HDFS">
      <component name="NAMENODE">
        <upgrade>
          <task xsi:type="restart-task"/>
        </upgrade>
      </component>

to

    <service name="HDFS">
      <component name="NAMENODE">
        <upgrade>
          <task xsi:type="restart-task" timeout-config="upgrade.parameter.nn-restart.timeout"/>
        </upgrade>
      </component>

Step 3: Add this to ambari.properties

upgrade.parameter.nn-restart.timeout=XXXXXX

where XXXXXX is the time in seconds

Step 4: Restart Ambari Server

Step 5: Now you can move on to your upgrade process

[#2]

Great article @Dinesh Chitlangia

Just to add to the details mentioned in the blog here, the property ("upgrade.parameter.nn-restart.timeout") change during the upgrade may not take effect even if Ambari servers are restarted. So have the property in place before the upgrade starts.

[#3]

@kkanchu Thanks for pointing out! Updated the article.