Migrate one of three kudu master to a new server

[#1]

Hi all, 

We are going through the hassle of rebuilding our boxes with CentOS7 as there is no upgrade path from CentOS6. Now I have looked over these Kudu documentation to migrating to multiple kudu master but the problem is the first warning it says it is unsafe to do this so I guess my question would be can I take one master making it a 2 node master then follow the documentation on migrating to a 3 node master setup. what process would I follow to remove the 3rd master other than just deleteing the role as I can only imagine how kudu will handle this. Also we are running kudu 1.4

[#2 · Accepted solution]

The following step below should help you:

1. take a phyciscal backup of the master you are going to take down (in case something is going wrong with the migration).
2. stop doing any DDL to the masters to prevent data loss. 
3. take down the master you chosen on step 1.
4. follow the doc Recovering from a Dead Kudu Master in a Multi-Master Deployment.

[#3]

Thanks Hoa Hoa, for the reply I will test these steps out they seem pretty straight forwards thanks for you help. One thing I have a question about is step 2 you are stay to stop doing any DDL to the masters does this mean I should take down the other masters or make sure we dont have any DDL stametns running which may be a little tricky once we move to production. 

[#4]

You don't need to take down the other masters, but need to establish a maintenance window (no DDLs) to ensure the table metadata is consistent between followers and the leader. If it is hard in production, alternative is to ensure the master to be taken down is not the leader (to avoid lose data).