Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera's Enterprise Data Hub (EDH) on the AWS Cloud

Solved Go to solution

Cloudera's Enterprise Data Hub (EDH) on the AWS Cloud

Rising Star

http://docs.aws.amazon.com/quickstart/latest/cloudera/welcome.html

 

Hey guys

 

Will this work for the starving developers version as well ? 

I am using CDH 5.8.0 

 

thanks

 

sanjay

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Cloudera's Enterprise Data Hub (EDH) on the AWS Cloud

Expert Contributor
Sorry, realized something -- Director installs CM/CDH 5.7 by default. In order to install 5.8, you'll need to modify the Cloudera Manager repository and CDH parcel repository to point to the most recent versions. The documentation for this can be found here: http://www.cloudera.com/documentation/director/latest/topics/director_get_started_aws_install_cm_cdh.... If you plan on using the command line interface, you'll be able to find the corresponding configuration options here: https://github.com/cloudera/director-scripts/blob/master/configs/aws.reference.conf.

View solution in original post

3 REPLIES 3
Highlighted

Re: Cloudera's Enterprise Data Hub (EDH) on the AWS Cloud

Expert Contributor

Hi Sanjay! If you're asking if you need an enterprise license to use Director, you do not. You should be able to use AWS Quickstart to deploy a functioning CDH 5.8.0 cluster without a license.

Highlighted

Re: Cloudera's Enterprise Data Hub (EDH) on the AWS Cloud

Expert Contributor
Sorry, realized something -- Director installs CM/CDH 5.7 by default. In order to install 5.8, you'll need to modify the Cloudera Manager repository and CDH parcel repository to point to the most recent versions. The documentation for this can be found here: http://www.cloudera.com/documentation/director/latest/topics/director_get_started_aws_install_cm_cdh.... If you plan on using the command line interface, you'll be able to find the corresponding configuration options here: https://github.com/cloudera/director-scripts/blob/master/configs/aws.reference.conf.

View solution in original post

Highlighted

Re: Cloudera's Enterprise Data Hub (EDH) on the AWS Cloud

Rising Star

Thats awesome thanks a ton Mike. Early this morning before your mails came in - I grew impatient :-)  as is my nature - and did give it a shot to the Cloudera director as-is-wher-is scripts.

 

1. Used the Cloud Formation template here

    https://s3.amazonaws.com/quickstart-reference/cloudera/latest/templates/Template2-Cloudera-AWS-Exist...

 

2. Created a "ClusterLauncher Instance" on AWS

 

3. SSH to "ClusterLauncher Instance"

    cloudera-director bootstrap cloudera/setup-default/aws.reference.conf 

Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Cloudera Director 2.1.0 initializing ...
The configuration file aws.reference.conf is not present or cannot be read.
[ec2-user@ip-10-219-178-74 ~]$ cloudera-director bootstrap cloudera/setup-default/aws.reference.conf
Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Cloudera Director 2.1.0 initializing ...
Installing Cloudera Manager ...
* Starting ..... done
* Requesting an instance for Cloudera Manager .......................... done
* Installing screen package (1/1) ....... done
* Running custom bootstrap script on [10.219.177.189, ip-10-219-177-189.us-west-2.compute.internal, 52.43.22.181, ec2-52-43-22-181.us-west-2.compute.amazonaws.com] .......... done
* Waiting for SSH access to [10.219.177.189, ip-10-219-177-189.us-west-2.compute.internal, 52.43.22.181, ec2-52-43-22-181.us-west-2.compute.amazonaws.com], default port 22 ..... done
* Inspecting capabilities of 10.219.177.189 .......... done
* Normalizing a3508870-3de7-4dc0-84a5-f69c77610c89 ..... done
* Installing ntp package (1/4) ..... done
* Installing curl package (2/4) ..... done
* Installing nscd package (3/4) ..... done
* Installing gdisk package (4/4) ..................... done
* Resizing instance root partition ......... done
* Mounting all instance disk drives ............ done
* Waiting for new external database servers to start running ........ done
* Installing repositories for Cloudera Manager ....... done
* Installing oracle-j2sdk1.7 package (1/3) ..... done
* Installing cloudera-manager-daemons package (2/3) ..... done
* Installing cloudera-manager-server package (3/3) ...... done
* Setting up embedded PostgreSQL database for Cloudera Manager ...... done
* Installing cloudera-manager-server-db-2 package (1/1) ..... done
* Starting embedded PostgreSQL database ...... done
* Starting Cloudera Manager server ... done
* Waiting for Cloudera Manager server to start ..... done
* Setting Cloudera Manager License ... done
* Enabling Enterprise Trial ... done
* Configuring Cloudera Manager ..... done
* Deploying Cloudera Manager agent ...... done
* Waiting for Cloudera Manager to deploy agent on 10.219.177.189 ... done
* Setting up Cloudera Management Services ............ done
* Backing up Cloudera Manager Server configuration ...... done
* Inspecting capabilities of 10.219.177.189 ...... done
* Done ...
Cloudera Manager ready.
Creating cluster C5-Reference-AWS ...
* Starting ..... done
* Requesting 11 instance(s) in 3 group(s) ....................................... done
* Preparing instances in parallel (20 at a time) .............................................................. done
* Waiting for Cloudera Manager installation to complete ... done
* Installing Cloudera Manager agents on all instances in parallel (20 at a time) ........ done
* Waiting for new external database servers to start running ... done
* Creating CDH5 cluster using the new instances ... done
* Creating cluster: C5-Reference-AWS .... done
* Downloading parcels: CDH-5.7.2-1.cdh5.7.2.p0.18,KAFKA-2.0.2-1.2.0.2.p0.5 ... done
* Distributing parcels: KAFKA-2.0.2-1.2.0.2.p0.5,CDH-5.7.2-1.cdh5.7.2.p0.18 ... done
* Activating parcels: KAFKA-2.0.2-1.2.0.2.p0.5,CDH-5.7.2-1.cdh5.7.2.p0.18 ...... done
* Configuring Hive to use Sentry ... done
* Creating Sentry Database ... done
* Calling firstRun on cluster C5-Reference-AWS ... done
* Waiting for firstRun on cluster C5-Reference-AWS .... done
* Running cluster post creation scripts ...... done
* Adjusting health thresholds to take into account optional instances. ... done
* Done ...

 

Don't have an account?
Coming from Hortonworks? Activate your account here