Question regarding client installation/deletions on datanodes

[#1]

Hello All,

We have a cluster with 160+ nodes. We are planning for an rolling upgrade from HDP 2.5 to 2.6 using ambari. We have edgenodes where all the clients has been installed. In addition, we also have clients installed on all the datanodes (150+ nodes) like HDFS client, Zookeeper client etc.

If you review the below link, we could see that ambari does validate the services multiple times by running service checks during upgrade process.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/upgrading_HDP_per...

This will consume a lot of time during the upgrade. We can minimize the time by limiting the clients service checks.

Do we really need all the clients to be installed on the datanodes? If we remove the clients from datanodes, will there be any impact on the cluster in terms of functionality. If there is no impact then what all clients can be removed .

Appreciate your input!!

[#2]

We are now sure that we can't remove the clients from Datanodes as our applications will start failing.

So, is there any way we can skip the service checks during the rolling upgrade process.

[#3 · Accepted solution]

@Shyam Shaw,

Note: I haven't tried it myself but hope this should work. Please be cautious if you trying it on production cluster

1) Find the upgrade xml file on ambari server host.

/var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml

2) Comment out the following piece of code in the upgrade XML file and save it. This piece of code will occur thrice . SERVICE_CHECK_1, SERVICE_CHECK_2, SERVICE_CHECK_3. Comment all the 3.

<group name="SERVICE_CHECK_1" title="All Service Checks" xsi:type="service-check">
      <direction>UPGRADE</direction>
      <skippable>true</skippable>
      <priority>
        <service>ZOOKEEPER</service>
        <service>RANGER</service>
        <service>RANGER_KMS</service>
        <service>HDFS</service>
        <service>KAFKA</service>
        <service>YARN</service>
        <service>MAPREDUCE2</service>
        <service>HBASE</service>
      </priority>
    </group>

3): skip service check failures. Change it to true from false

<skip-service-check-failures>true</skip-service-check-failures> 

4) Restart Ambari Server for it to pick up the changes.

ambari-server restart

5) Start rolling upgrade

Thanks,

Aditya

[#4]

@Aditya Sirna

Thanks for the response. Could there be any downside of skipping the service check during rolling upgrade.

[#5]

@Shyam Shaw,

There should not be any issue because of this. Please check Error Handling in the doc

https://community.hortonworks.com/articles/2473/rolling-upgrade-express-upgrade-in-ambari.html

[#6]

@Aditya Sirna

Thanks for the information.