For now. I think the two possible methods I outlined might work. Additionally, you could export the data to something like parquet or avro, using Spark or Impala, and then reload the data in the new cluster. ... View more

Kudu doesn't support swapping a drive to a new host.   Kudu tablet servers store the consensus configuration of their tablets, and the master also stores the consensus configuration for the tablets. By moving all the servers, you changed all the hostnames, and now the cluster is in total disarray. It's possible to rewrite the consensus configuration of the tablets on the tablet servers, but I'm not sure there's currently a way to rewrite the data in the master. So, by scripting `kudu local_replica cmeta rewrite_raft_config` you could fix the tablet servers. You will need to rewrite the config of each tablet so the hostnames are mapped from the old servers to the new servers. If you do that correcty and the tablet replicas are able to elect a leader, the leader will send updated information to the master, which should cause it to update its record. I don't think too many people have ever tried anything like this, so there may be other things that need to be fixed, or it simply might not be possible to recover the cluster.   What you should have done is set up the new cluster, then transferred the data via Spark or an Impala CTAS statement, or you should have built the new cluster as an expansion of the existing one, and then decommissioned all the tablet servers of the old cluster, and then moved the master nodes one-by-one to the new cluster. ... View more

This is a known issue with some code to auto-detect whether replicas of non-replicated tablets can be moved without issues (see KUDU-2443). The code relied on std::regex. The tool was built with g++/libstdc++ of versions < 4.9, which means std::regex unexpectedly fails to compile a regular expression containing a bracket, throwing a std::regex_error exception (see [1]). Starting from version 4.9.1, the libstdc++ has proper support for the C++11's regular expressions (see [2]). This makes the kudu CLI crash if running 'kudu cluster rebalance' on the following platforms: * RHEL/CentOS 7 * Ubuntu14.04 LTS (Trusty) * SLES12   You should be able to work around the problem by specifying the flag --move_single_replicas to either 'enabled' or 'disabled', as you require, instead of the default 'auto'.   Unfortunately there's no release in the CDH 5 line in which this issue is fixed (yet). ... View more