As @Grizzly indicated, I would think you would do this in two phases: 1) install CM over your existing CDH cluster (see this thread), then 2) upgrade via CM to CDH5. ... View more

You can use the API to stop and then subsequently start the cluster programatically if you'd like.   The one question I have is why you feel it is required to restart all the services in between test runs.  Is it because you want to make sure heap is cleared and just assure there are no lingering file handles/etc. from the last job? ... View more

Frequent bulk loads is not a good long-term use case for HBase...for the exact reasons you mentioned.  The compaction queue and data maintenance overhead just never catches up.   Can you either A) just stream the writes into HBase constantly using puts, or B) hold off on the bulk loads until the HFiles are full 18GB (eg. one region) in size?  At least reduce the compactions and splits? ... View more

@vkii I believe this thread will guide you to the general answer to your question.  It is a painstaking and risky process, but can be accomplished with careful planning.   Clint ... View more

It is true that a restart would be required at some point along the way if you tried to overlay Cloudera Manager on top of an existing CDH cluster.  You would also need to take very careful steps to assure that all your configs were ported into CM during the install process, it's not really a documented process because it can cause data loss if you don't know what you're doing.  see this thread   Cloudera Manager is not just a monitoring tool, it is an entire management suite and wants to install, configure, monitor, and report on your cluster.  Just keep in mind that CM will move your configs around and wants to manage all those settings and keep it's own log directories, etc.  It will not drop in seemlessly into an existing suite of tools that are managing the cluster and only do the monitoring piece for you.   HTH ... View more

You may find it much more difficult to attempt to overlay Cloudera Manager on top of an existing CDH cluster rather than just starting over and letting Cloudera Manager install the CDH software for you.  The Cloudera Manager Installation process, documented here, actually installs CDH and configures it for you in our recommended "best practices" configuration.  It's a very simple process through CM, I would recommend you do that. ... View more

You shouldn't have to disable the Master service at all.  If you have two HBase masters running (one Active, one Backup), simply stopping the service on the Active Master will trigger the Backup master to take over.  The graceful transition details are handled internally and very effectively by HBase itself. ... View more