How to use CM API to distribute parcel to a host

[#1]

I am creating a python script which is based on CM API to add hosts to a cluster.

My challenges is:

I couldn't find which API call to distribute parcel to hosts especially if there is a way to install the parcel on specified hosts. I checked the following link but didn't find anything useful:

https://cloudera.github.io/cm_api/epydoc/5.8.0/cm_api-module.html

[#2 · Accepted solution]

CM will automatically de-activate conflicting parcels, which is why you didn't need to de-activate 5.5.1 first. It would be a burden on the user to identify and de-activate all conflicting parcels, and this is by design.

The CM UI will show the upgrade option when you are trying to activate a CDH parcel of a higher version. This will take you through the upgrade wizard, which is generally what you want.

In your case, you don't really want the upgrade wizard and can use the API to directly activate 5.5.1 again.

If your API script should never be run when CDH is already installed, you can add a check to your script to validate that requirement.

Rather than trying to test with a single host within a cluster, you can make a test cluster with a single host and try out your API script.

[#3]

Actually, I found some examples here:

http://cloudera.github.io/cm_api/docs/python-client/#managing_parcels

But the problem in those examples is that the API download/distribute/activate the parcel on all cluster nodes, I am not feeling comfortable doing that as if the script by accident specified a wrong parcel, it will screw up the entire cluster, is there a way to download the parcel to a certain host only?

Thanks!

[#4]

Hi Keagles,

CM always distributes parcels to the entire cluster. It's generally much easier to manage your cluster when all hosts are running the same software, and the overhead of distributing and storing these binaries is generally very small compared to the real cluster activity and size of your real data.

You can easily deactivate, undistribute, and delete parcels if you don't like the change you made. You can also test things out by just downloading the parcel via the API, then checking in the CM UI that you got the right one.

[#5]

Thanks Darren.

Our CM server has two parcels installed-- cdh4.7 and cdh5.5.1. Currently cdh5.5.1 is active. 

By doing a test on a test cluster, I found I can directly activate cdh4.7 parcel without deactivate cdh5.5.1 first, and after that cdh 5.5.1 only shows an upgrade option( I cannot activate it). To re-activate cdh5.5.1 parcel, I have to go through the cluster upgrade wizard again and of course the wizard would fail in HDFS metadata upgrade step as the cluster had already been upgraded to CDH5.

Although I can do some safety guard such as make sure the  parcel version is 5.5.1 before distribute and activate it, remove chd old cdh4.7 parcel etc. But I still feel not very comfortable to use the parcel API call for the entire cluster.

I hope Cloudera can improve the parcel management by doing one or both of the following:

1: Before activate a new parcel, the current parcel need to be deactivate first.

2: Instead of pushing a parcel to the entire cluser, admin has option to push to certain host. I know the cluster should have the same parcel on every host, but I hope at least make this option optional. CM can show a warning flag if not all hosts run the same parcel.

[#6 · Accepted solution]

CM will automatically de-activate conflicting parcels, which is why you didn't need to de-activate 5.5.1 first. It would be a burden on the user to identify and de-activate all conflicting parcels, and this is by design.

The CM UI will show the upgrade option when you are trying to activate a CDH parcel of a higher version. This will take you through the upgrade wizard, which is generally what you want.

In your case, you don't really want the upgrade wizard and can use the API to directly activate 5.5.1 again.

If your API script should never be run when CDH is already installed, you can add a check to your script to validate that requirement.

Rather than trying to test with a single host within a cluster, you can make a test cluster with a single host and try out your API script.

[#7]

Thanks! So in my script, I just need to run distribute and active the parcel, I don't need to download it. Is that what you mean?

Thanks again!

[#8]

Sorry I mean just distribute the parcel without download or activate it. Is that right? This is becuase the parcle is already downloaded and activated.

[#9]

Read about parcel lifecycles here:
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_parcels.html#concept_k1c_wgx_p...

All activated parcels are already downloaded and distributed.

[#10]

The question is that to add a new host to the cluster. Since the parcle is already downloaded, distributed and activated cluster wise, do I need to at least run distribute for this host? or I just run create_host() call and parcels will be distributed and activated automaitcally on this host?

[#11 · Accepted solution]

Oh sorry, I misunderstood.

When you add a host, CM will automatically distribute all parcels to that host and activate it so it matches all other hosts in the cluster.