Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)

During upgrade, if Namenode restarts timeout, it may not appear to be a problem as the request times out from the Ambari UI but the restart process continue to run in the background.

However, this can lead to inconsistencies in Ambari database and cause further issues at Finalize upgrade step.

Note: This article is only useful upto Ambari-2.5.x version and must be performed before starting the upgrade process.

Ambari-2.6 onwards it is a one step change where you only need to update the ambari.properties file instead of all the xml changes listed below.

For Ambari-2.6 onwards: refer this document https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/preparing_to_upgr...

Caution: The steps described below are a hack! It is not recommended that we go about making changes to upgrade XML files based on our needs. Please exercise caution and measure your risks before following the steps.

You can increase the timeouts for namenode restart using the following steps before you start the upgrade process:

Step 1: Locate the upgrade file on Ambari server host. If we are upgrading from HDP-2.5 to HDP-2.6 then :

/var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml [Express Upgrade]

/var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml [Rolling Upgrade]

Step 2: Change

    <service name="HDFS">
      <component name="NAMENODE">
        <upgrade>
          <task xsi:type="restart-task"/>
        </upgrade>
      </component>

to

    <service name="HDFS">
      <component name="NAMENODE">
        <upgrade>
          <task xsi:type="restart-task" timeout-config="upgrade.parameter.nn-restart.timeout"/>
        </upgrade>
      </component>

Step 3: Add this to ambari.properties

upgrade.parameter.nn-restart.timeout=XXXXXX

where XXXXXX is the time in seconds

Step 4: Restart Ambari Server

Step 5: Now you can move on to your upgrade process

911 Views
Comments
New Contributor

Great article @Dinesh Chitlangia

Just to add to the details mentioned in the blog here, the property ("upgrade.parameter.nn-restart.timeout") change during the upgrade may not take effect even if Ambari servers are restarted. So have the property in place before the upgrade starts.

@kkanchu Thanks for pointing out! Updated the article.

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎11-09-2017 03:23 AM
Updated by:
 
Contributors
Top Kudoed Authors