Install CDH5 on EC2 without human interaction

[#1]

Hi all!

At the company I work, we're currently using a 4 node Amazon EMR cluster together with S3 for all our data warehousing and analysis needs. The cluster gets spin-up each morning and torn down each evening automatically through a cron job running on another server, to save costs.

We're using Impala exstensively. Our data is copied each morning from S3 to HDFS after the cluster has been spun up.

I was looking at installing Hue to provide a nice interface for querying Impala. Then it occurred to me that it would probably be easier to move from EMR to EC2 and install CDH5 on there. Ideally we would use Cloudera Manager for monitoring the cluster while it's running.

The problem: is there a way to install CDH5, including Cloudera Manager, automatically on an EC2 cluster, without human interaction?

[#2]

The Cloudera Manager and CDH installation can be automated though it would
require a bit of programming from your side:

Read about Path B installation in this link:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installat...

Then you can use the Cloudera Manager API to add hosts, services and roles
to your cluster
http://cloudera.github.io/cm_api/

[#3]

I forgot to add to my earlier note, you can also try using Apache Whirr to provision clusters on AWS

http://blog.cloudera.com/blog/2012/10/set-up-a-hadoophbase-cluster-on-ec2-in-about-an-hour/

[#4]

Thanks for the swift answer! I have looked at the API and it seems you can't actually install packages through the API, right? Does that mean that all the packages for all the services I'd want to enable, should be installed beforehand on all nodes, before I add hosts, services and roles through the API?

[#5]

Yes the rpm/deb packages have to be installed already. Alternatively you
could use a mixture of the AWS API (to provision the hosts), then use the
Cloudera Manager API to provision the cluster (using the parcel deployment)

[#6]

---

@GautamG wrote:
Yes the rpm/deb packages have to be installed already. Alternatively you
could use a mixture of the AWS API (to provision the hosts), then use the
Cloudera Manager API to provision the cluster (using the parcel deployment)

---

Does the CM API support distributing parcels? Or how would I go about that? I know how to provision EC2 instances using the Amazon AWS API, but now I'm kind off in the dark on how to install CM and CDH on those 🙂

Regarding the Whirr option: it doesn't support YARN on EC2 with CM yet, right?

[#7]

Please refer to the Path B install link I provided earlier which explains how you can automate CM and CDH installation. Then refer to the CM API (http://cloudera.github.io/cm_api/apidocs/v7/rest.html) specifically the /clusters/{clusterName}/parcels mountpoints.

[#8]

The Cloudera Documentation refers to deploying a YARN cluster on AWS

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/c...

[#9]

That's interesting, because this page in the CDH5 documentation states:

Note: At present you can launch and run only an MapReduce cluster; YARN is not supported.

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installat...

[#10 · Accepted solution]

I have to check on that. Meanwhile another option is to create the cluster
manually and save the master and worker node images as custom AMIs. Use
those AMIs every morning to create a new cluster, then tear it down. When
you want to update CDH, just do it once manually and save new AMIs