You should be able to reinstall the packages via yum and be back in business.  Your data (at least HDFS) should not have been removed in this process.  yum will do this if the package you are trying to remove has dependencies.  I believe there are command-line options to tell it to forget about the dependencies. ... View more

Marc,     These are difficult challenges, to be sure, but I will attempt to address your concerns here:   1) The theoretic limit to how large your replication queue can grow is the size of your HDFS storage, since the items needing replication are stored in HBase WALs on HDFS.  I'm not aware of any real world tests that have been done to prove that limit, though.   In your case it sounds like your replication queue would definitely grow very large very rapidly, so this is something to take into consideration.  For an extended outage like you describe (eg. weeks) you might actually be better off just disabling replication on the surviving cluster and start over from scratch once you have restored the downed cluster.  In other words, restore a fresh copy of your data to the new cluster and then enable replication again.  Snapshots would be a good way to reasonably get a current image copied to the restored cluster.  More on that below...   2) The backup and disaster recovery options I described in the blog do take "restore" into account and provide functionality for that, but they largely leave the implementation up to you.  In other words, you will have to come up with a gameplan that works for your environment/infrastructure and test it.  As I alluded above, I might recommend that you disable replication on the surviving cluster for an extended outage on the order of weeks.  Once a cluster is available for replication again, you could export a snapshot of your existing cluster and use bulk loads to load the data into the new cluster.  This can be done on a per-table basis, so you can choose your most critical tables first and enable replication on them at the same time in order to get your data mostly in sync.  Note that there might still be some data that gets out of sync since your cluster is so active, this would require a bit of manual intervention to add/update the rows that are not in sync.  However, once they are sync'd up, replication will take over again and you'll be fine.  As you indicated, the CopyTable and Export functions would not be preferable as they would put a heavy MapReduce and HBase API load on your source cluster.  Snapshots do not introduce such a load during their creation and although running "exportSnapshot" (or even using DistCP to copy the resulting snapshot to the restored cluster) will create a Mapreduce job, that job will not burden HBase as much as it will not make API calls to the tables at all.   3) As previously stated, I think your best bet here is to restore the most active tables first.  Since you don't want to make the data in the restored cluster available for client requests until it's all sync'd up with the surviving cluster anyway (for data accuracy sake), it's really a moot point to attempt to restore data in a reverse order.  Nonetheless, no such functionality exists that I'm aware of.   For an interesting read, see how Facebook did it. ... View more

Yes, DistCP is usually what people use for that.  It has rudimentary functionality for sync'ing data between clusters, albeit in a very busy cluster where files are being deleted/added frequently and/or other data is changing, replicating those changes between clusters will require custom logic on top of HDFS.  Facebook developed their own replication layer, but it is proprietary to their engineering department. ... View more

I'll take a stab at addressing these questions:   1) Yes, you will need to shut down all hadoop services on all nodes before you perform a move like this, because HDFS will naturally attempt to re-replicate all the data that was residing on the 10 datanodes which you shut down.  And since that would be half your cluster, it's likely that there would be some blocks that could not be re-replicated because the only copies of those blocks resided on the 10 nodes you shut down, so your HDFS would go into safe mode due to under-replicated/missing blocks.  No risk of data loss, just not the way you'd like to do it.   2) If you properly shut down all services before doing the move, there is no risk of data loss.  Just be sure your move doesn't entail giving the machines new IP addresses/hostnames, as this is an entirely different operation that requires a careful migration process.     3) yes   4) as stated in my response #1, you will get data replication churn in your cluster if you shut down individual datanodes.  Cloudera Manager (enterprise) supports the notion of a rolling restart for your services if you'd like to maximize uptime, but otherwise you'll get the Namenode trying to re-replicate data if you stop one single node.  After a certain timeout is reached, at least.  I think you have several minutes before the blocks will begin to re-replicate to other nodes. ... View more

OK, it sounds like your yum repo is messed up.  So here are the steps to get yum working and also pointing to the CDH4.4.0 packages:   1) download this repo file and place it in /etc/yum.repos.d  (you should move your existing cloudera-cdh4.repo file to some other location for backup)   2) edit the file and modify the "baseurl" property to point to this URL:   http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/4.4.0/   3) Now you should be able to use these instructions to install CDH4.4.0 by just doing "yum install" commands manually (Note, you'll have to scroll down the page to the section titled "Step 2: Install CDH4 with MRv1" and there you should see the yum commands that will work for you.   Please let me know if any other questions/concerns arise.  Installing CDH manually is a little more involved that's why the documentation is so long.   ... View more

Thanks for the feedback, I will file a JIRA with our docs team to have this updated.   Much appreciated! ... View more