question Re: Install CDH5 on EC2 without human interaction in Archives of Support Questions (Read Only)

Install CDH5 on EC2 without human interaction

Daan — Tue, 21 Apr 2026 14:00:56 GMT

Hi all!

At the company I work, we're currently using a 4 node Amazon EMR cluster together with S3 for all our data warehousing and analysis needs. The cluster gets spin-up each morning and torn down each evening automatically through a cron job running on another server, to save costs.

We're using Impala exstensively. Our data is copied each morning from S3 to HDFS after the cluster has been spun up.

I was looking at installing Hue to provide a nice interface for querying Impala. Then it occurred to me that it would probably be easier to move from EMR to EC2 and install CDH5 on there. Ideally we would use Cloudera Manager for monitoring the cluster while it's running.

The problem: is there a way to install CDH5, including Cloudera Manager, automatically on an EC2 cluster, without human interaction?

Re: Install CDH5 on EC2 without human interaction

GautamG — Mon, 01 Sep 2014 23:14:35 GMT

The Cloudera Manager and CDH installation can be automated though it would
require a bit of programming from your side:

Read about Path B installation in this link:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installation-Guide/cm5ig_install_cm_cdh.html

Then you can use the Cloudera Manager API to add hosts, services and roles
to your cluster
http://cloudera.github.io/cm_api/

Re: Install CDH5 on EC2 without human interaction

GautamG — Tue, 02 Sep 2014 09:12:38 GMT

I forgot to add to my earlier note, you can also try using Apache Whirr to provision clusters on AWS

http://blog.cloudera.com/blog/2012/10/set-up-a-hadoophbase-cluster-on-ec2-in-about-an-hour/

Re: Install CDH5 on EC2 without human interaction

Daan — Tue, 02 Sep 2014 09:18:23 GMT

Thanks for the swift answer! I have looked at the API and it seems you can't actually install packages through the API, right? Does that mean that all the packages for all the services I'd want to enable, should be installed beforehand on all nodes, before I add hosts, services and roles through the API?

Re: Install CDH5 on EC2 without human interaction

GautamG — Tue, 02 Sep 2014 09:42:35 GMT

Yes the rpm/deb packages have to be installed already. Alternatively you
could use a mixture of the AWS API (to provision the hosts), then use the
Cloudera Manager API to provision the cluster (using the parcel deployment)

Re: Install CDH5 on EC2 without human interaction

Daan — Tue, 02 Sep 2014 09:49:15 GMT

@GautamG wrote:
Yes the rpm/deb packages have to be installed already. Alternatively you
could use a mixture of the AWS API (to provision the hosts), then use the
Cloudera Manager API to provision the cluster (using the parcel deployment)

Does the CM API support distributing parcels? Or how would I go about that? I know how to provision EC2 instances using the Amazon AWS API, but now I'm kind off in the dark on how to install CM and CDH on those 🙂

Regarding the Whirr option: it doesn't support YARN on EC2 with CM yet, right?

Re: Install CDH5 on EC2 without human interaction

GautamG — Tue, 02 Sep 2014 09:56:22 GMT

Please refer to the Path B install link I provided earlier which explains how you can automate CM and CDH installation. Then refer to the CM API (http://cloudera.github.io/cm_api/apidocs/v7/rest.html) specifically the /clusters/{clusterName}/parcels mountpoints.

Re: Install CDH5 on EC2 without human interaction

GautamG — Tue, 02 Sep 2014 09:59:35 GMT

The Cloudera Documentation refers to deploying a YARN cluster on AWS

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_whirr_define.html

Re: Install CDH5 on EC2 without human interaction

Daan — Tue, 02 Sep 2014 10:01:26 GMT

That's interesting, because this page in the CDH5 documentation states:

Note: At present you can launch and run only an MapReduce cluster; YARN is not supported.

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installation-Guide/cm5ig_launch_cm_with_whirr.html

Re: Install CDH5 on EC2 without human interaction

GautamG — Tue, 02 Sep 2014 10:13:35 GMT

I have to check on that. Meanwhile another option is to create the cluster
manually and save the master and worker node images as custom AMIs. Use
those AMIs every morning to create a new cluster, then tear it down. When
you want to update CDH, just do it once manually and save new AMIs

Re: Install CDH5 on EC2 without human interaction

Daan — Tue, 02 Sep 2014 10:22:08 GMT

@GautamG wrote:
I have to check on that. Meanwhile another option is to create the cluster
manually and save the master and worker node images as custom AMIs. Use
those AMIs every morning to create a new cluster, then tear it down. When
you want to update CDH, just do it once manually and save new AMIs

Hmmm, that is actually a great idea! It certainly is the least-effort solution so far 🙂 I will look into that this afternoon.

Meanwhile, with regards to Whirr, it seems the documentation I pointed to, is outdated. If you look at the sample whirr config in the whirr-cm repo, it supports YARN roles: https://raw.githubusercontent.com/cloudera/whirr-cm/master/cm-ec2.properties

Re: Install CDH5 on EC2 without human interaction

GautamG — Tue, 02 Sep 2014 10:24:35 GMT

Thank you for the feedback.

Re: Install CDH5 on EC2 without human interaction

Daan — Tue, 02 Sep 2014 10:25:13 GMT

Thank you for your help!

Re: Install CDH5 on EC2 without human interaction

GautamG — Wed, 15 Oct 2014 01:55:41 GMT

You might be pleased to know that a new product called Cloudera Director has been released that enables you to quickly spin up clusters on Cloud platforms like AWS. Amazon also supports the creation of EDH clusters within AWS QuickStart, this uses Cloudera Director in the background. You can read more in the following links:

http://aws.amazon.com/quickstart/

http://www.cloudera.com/content/cloudera/en/products-and-services/director.html

Re: Install CDH5 on EC2 without human interaction

Daan — Wed, 15 Oct 2014 10:13:19 GMT

I noticed! Very cool development!