I am currently working on a cluster migration from HDP 2.2 to HDP 2.6. All in all I will have to migrate 15 different clusters so I am thinking of automation. I know blueprints, which can be used for an initial cluster creation, but not for cluster modifications. So I am finally not sure how to do the migrations.
Here is what I need to do :
As you can read, my goal is to have all clusters upgraded with the "correct" configuration. I have basically two kind of clusters (so I can reuse the configurations) (small and large).
What do you think ?
Blueprint is a cluster creation template that is good for automating the cluster creation.
However if you already have 15 cluster created and want to upgrade them to HDP 2.6 then the i see the only approach will be to perform the Rolling Upgrade / Express Upgrade as it will take care of a lots of things like taking care of Schema Upgrade for NameNode / Hive Metastore ...etc.
The best approach for the upgrade will be to use the Rolling Upgrade / Express Upgrade options from Ambari UI. With the ambari managed upgrade we get a better option to pause the upgrade when needed and also to finalize the upgrade or rollback in case of any issue.
But how (in case of an Express or Rolling Upgrade) I should handle the configuration upgrade? I forget to mention that we have customized some properties (for example : "hadoop-env template"). We added for example JMX for all namenodes/datanodes and hbase etc. In this case, Ambari won't upgrade the properties since they have been modified. An here comes the problem: As the properties (hadoop_env, hbase_env...) have changed to use Java 8, how should I handle mine?
Here is what I could do:
If I have to do this for all the 15 clusters, this will take some time. So I am searching for automation as well.