Cloudera's Enterprise Data Hub (EDH) on the AWS Cloud

[#1]

http://docs.aws.amazon.com/quickstart/latest/cloudera/welcome.html

Hey guys

Will this work for the starving developers version as well ? 

I am using CDH 5.8.0 

thanks

sanjay

[#2]

Hi Sanjay! If you're asking if you need an enterprise license to use Director, you do not. You should be able to use AWS Quickstart to deploy a functioning CDH 5.8.0 cluster without a license.

[#3 · Accepted solution]

Sorry, realized something -- Director installs CM/CDH 5.7 by default. In order to install 5.8, you'll need to modify the Cloudera Manager repository and CDH parcel repository to point to the most recent versions. The documentation for this can be found here: http://www.cloudera.com/documentation/director/latest/topics/director_get_started_aws_install_cm_cdh.... If you plan on using the command line interface, you'll be able to find the corresponding configuration options here: https://github.com/cloudera/director-scripts/blob/master/configs/aws.reference.conf.

[#4]

Thats awesome thanks a ton Mike. Early this morning before your mails came in - I grew impatient 🙂  as is my nature - and did give it a shot to the Cloudera director as-is-wher-is scripts.

1. Used the Cloud Formation template here

    https://s3.amazonaws.com/quickstart-reference/cloudera/latest/templates/Template2-Cloudera-AWS-Exist...

2. Created a "ClusterLauncher Instance" on AWS

3. SSH to "ClusterLauncher Instance"

    cloudera-director bootstrap cloudera/setup-default/aws.reference.conf 

Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Cloudera Director 2.1.0 initializing ...
The configuration file aws.reference.conf is not present or cannot be read.
[ec2-user@ip-10-219-178-74 ~]$ cloudera-director bootstrap cloudera/setup-default/aws.reference.conf
Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Cloudera Director 2.1.0 initializing ...
Installing Cloudera Manager ...
* Starting ..... done
* Requesting an instance for Cloudera Manager .......................... done
* Installing screen package (1/1) ....... done
* Running custom bootstrap script on [10.219.177.189, ip-10-219-177-189.us-west-2.compute.internal, 52.43.22.181, ec2-52-43-22-181.us-west-2.compute.amazonaws.com] .......... done
* Waiting for SSH access to [10.219.177.189, ip-10-219-177-189.us-west-2.compute.internal, 52.43.22.181, ec2-52-43-22-181.us-west-2.compute.amazonaws.com], default port 22 ..... done
* Inspecting capabilities of 10.219.177.189 .......... done
* Normalizing a3508870-3de7-4dc0-84a5-f69c77610c89 ..... done
* Installing ntp package (1/4) ..... done
* Installing curl package (2/4) ..... done
* Installing nscd package (3/4) ..... done
* Installing gdisk package (4/4) ..................... done
* Resizing instance root partition ......... done
* Mounting all instance disk drives ............ done
* Waiting for new external database servers to start running ........ done
* Installing repositories for Cloudera Manager ....... done
* Installing oracle-j2sdk1.7 package (1/3) ..... done
* Installing cloudera-manager-daemons package (2/3) ..... done
* Installing cloudera-manager-server package (3/3) ...... done
* Setting up embedded PostgreSQL database for Cloudera Manager ...... done
* Installing cloudera-manager-server-db-2 package (1/1) ..... done
* Starting embedded PostgreSQL database ...... done
* Starting Cloudera Manager server ... done
* Waiting for Cloudera Manager server to start ..... done
* Setting Cloudera Manager License ... done
* Enabling Enterprise Trial ... done
* Configuring Cloudera Manager ..... done
* Deploying Cloudera Manager agent ...... done
* Waiting for Cloudera Manager to deploy agent on 10.219.177.189 ... done
* Setting up Cloudera Management Services ............ done
* Backing up Cloudera Manager Server configuration ...... done
* Inspecting capabilities of 10.219.177.189 ...... done
* Done ...
Cloudera Manager ready.
Creating cluster C5-Reference-AWS ...
* Starting ..... done
* Requesting 11 instance(s) in 3 group(s) ....................................... done
* Preparing instances in parallel (20 at a time) .............................................................. done
* Waiting for Cloudera Manager installation to complete ... done
* Installing Cloudera Manager agents on all instances in parallel (20 at a time) ........ done
* Waiting for new external database servers to start running ... done
* Creating CDH5 cluster using the new instances ... done
* Creating cluster: C5-Reference-AWS .... done
* Downloading parcels: CDH-5.7.2-1.cdh5.7.2.p0.18,KAFKA-2.0.2-1.2.0.2.p0.5 ... done
* Distributing parcels: KAFKA-2.0.2-1.2.0.2.p0.5,CDH-5.7.2-1.cdh5.7.2.p0.18 ... done
* Activating parcels: KAFKA-2.0.2-1.2.0.2.p0.5,CDH-5.7.2-1.cdh5.7.2.p0.18 ...... done
* Configuring Hive to use Sentry ... done
* Creating Sentry Database ... done
* Calling firstRun on cluster C5-Reference-AWS ... done
* Waiting for firstRun on cluster C5-Reference-AWS .... done
* Running cluster post creation scripts ...... done
* Adjusting health thresholds to take into account optional instances. ... done
* Done ...