Is there any best practice or guide to perform hardware(RAM) upgrade in live cluster ?

[#1 · Accepted solution]

While we do not provide a best practice to this I suggest the following: 1. Bring down the entire cluster cleanly and perform the RAM upgrade.

2. Use this to determine what all your new config settings should be:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/determi... (or use the correct one for your HDP release, this is from 2.6.0)

3. Once you change your memory settings, restart the cluster.

[#2]

If you want to minimize your down time you can try to stop nodes one by one, upgrade RAM and restart all components on the node after restart. You must do masters one by one, but in case of workers you can do in sets of 2, or if you have rack-awarness in sets of 3-4 or even rack by rack. You need HA configuration of major services like HDFS, Yarn, HBase, Hive for this to work. You also need replication of Kafka topics of at least 2, and do Kafka nodes one by one, if you have Kafka.