Reply
New Contributor
Posts: 7
Registered: ‎07-11-2014
Accepted Solution

"Non-destructive" Cloudera Manager install

Hi,

 

I tried a search on this, but the topic is a bit squishy:

 

I'm trying to find a good, non-destructive way to install Cloudera Manager and integrate an *existing* and working cluster. Said cluster was manually installed (for the learning experience), as per the CDH5 docs and the Hadoop Operations book.

 

Some first minor attempts, using one of the datanodes/nodemanagers as a guinea pig, scared me: CM tried to force install all sorts of packages, never checking whether and what was already there. Never making an attempt of reading existing config files and importing them first.

 

I have local repositories (for yum) all set up and working. I know how to point CM at them to use them.

 

However, I would like to be able to point CM at my exising nodes (namenodes/resourcemanagers, datanodes/nodemanager, etc) and integrate their existing config, adjusting what's needed, adding packages only when I expressly tell CM to.

 

Is there a safe way?

 

Thanks

Mike

Cloudera Employee
Posts: 228
Registered: ‎09-23-2013

Re: "Non-destructive" Cloudera Manager install

Mike,


Unfortunately there is no magic "ingest CDH only cluster" pre-made toolset that is part of the product.  The discrete XML configuration files you have set up for the raw CDH cluster, generally would need to be manually introduced through the CM configuration UI.  There was early work attempting to do this, but it was found to not be reliable across the wide variation 

 

We do have an API for cloudera manager that allows you to inspect and set cluster configuration values programatically (documented here http://cloudera.github.io/cm_api/apidocs/v6/ and here for examples and historical version reference http://cloudera.github.io/cm_api/).


When you consider what is generally "default" for the configuration, most of that is handled by CM, it would be specific non-default settings that you are using in your CDH cluster that would need to be brought over.  The API can be used to "dump" the current configuration of the CM database (where config ends up in a CM managed cluster) from a new install to look at what is set by default and compare from a test cluster.

 

THanks

Todd

 

 

New Contributor
Posts: 7
Registered: ‎07-11-2014

Re: "Non-destructive" Cloudera Manager install

Thanks, Todd (oh and, hi, to a former fellow Navidecer! Been a while... ;-)

 

I'm going to give the API a look, though first I'll try the config dump. I wonder if a staged migration would be possible. First, temporarily lower replication to a minimum, maybe two. Then start phasing out datanodes and letting them be marked as offline. Finally put those under CM control and migrate data from the other remaining nodes.

 

Need to have a think about this...

New Contributor
Posts: 7
Registered: ‎07-11-2014

Re: "Non-destructive" Cloudera Manager install


@Tgrayson wrote:

Mike,


[...]

 

We do have an API for cloudera manager that allows you to inspect and set cluster configuration values programatically (documented here http://cloudera.github.io/cm_api/apidocs/v6/ and here for examples and historical version reference http://cloudera.github.io/cm_api/).

 

[...]


Did those links go dead? Getting 404s now?

Posts: 416
Topics: 51
Kudos: 88
Solutions: 49
Registered: ‎06-26-2013

Re: "Non-destructive" Cloudera Manager install

I believe the community tool munged the hyperlinks when Todd posted the URLs, but the URLs are still correct:

 

http://cloudera.github.io/cm_api/apidocs/v6/

 

and

 

http://cloudera.github.io/cm_api/

Highlighted
New Contributor
Posts: 7
Registered: ‎07-11-2014

Re: "Non-destructive" Cloudera Manager install

Ah, those work. Thanks, guys!
Announcements