question Re: Pros and cons of using EMR vs Cloudbreak for launching hadoop cluster on AWS in Archives of Support Questions (Read Only)

Pros and cons of using EMR vs Cloudbreak for launching hadoop cluster on AWS

obaid_salikeen — Sun, 23 Oct 2016 15:04:38 GMT

Hi all,

I am a newbie to HDP and cloudbreak. I want to move some of our onsite Hadoop clusters/jobs on AWS. Two solutions that I have came-across are Cloudbreak and EMR, however not sure which one to use.

I wanted to know which technology to use for launching hadoop jobs on AWS? Pros and cons of using either approach would be really helpful (interms of cost, ease of use, monitoring, metrics, latency etc). One apparent cost optimization feature that I am interested in : is to launch the cluster whenever a job or jobs needs to run, and kill the cluster/nodes whenever there are no more jobs to execute.

Thanks

Obaid

Re: Pros and cons of using EMR vs Cloudbreak for launching hadoop cluster on AWS

Dominika — Tue, 25 Oct 2016 03:19:03 GMT

Hi @Obaid Salikeen, You may also consider using Hortonworks Data Cloud (currently in technical preview stage. See http://hortonworks.github.io/hdp-aws/.

Re: Pros and cons of using EMR vs Cloudbreak for launching hadoop cluster on AWS

obaid_salikeen — Tue, 25 Oct 2016 06:22:04 GMT

Thanks @Dominika B,

Thanks for sharing the link, seems interesting.

So I have a very basic question: Amazon EMR lets you launch manage Hadoop and Spark clusters, so what would be the benefit of using Hortonworks cloud vs just using EMR?

Thanks

Obaid

Re: Pros and cons of using EMR vs Cloudbreak for launching hadoop cluster on AWS

akanto — Wed, 26 Oct 2016 04:23:32 GMT

Hi @Obaid Salikeen,

Pros:

Multiple cloud provider support (ypu can deploy clusters using the same interface to different providers)
You can use it even on private cloud e.g OpenStack
Cloudbreak and HDP is open source
Cloudbreak installs Ambari, what you can use to monitor or customise your cluster after deployment (e.g. add new services)
It comes with fully configured SaltStack what you can use to manage your VMs e.g apply security patches
More flexible since you can create your own Blueprint which can contains only those services what you need
Cloudbreak supports autoscaling based on metrics gathered from Ambari (e.g some of those metrics are very general e.g. disk space others are Hadoop specific e.g. pending YARN containers)

Cons:

need one more instance where Cloudbreak is running (of course one Cloudbreak can manage multiple clusters)
Cloudbreak is a cluster management tool and you cannot submit jobs through it. Something like steps in EMR is not supported

Disclaimer: I am an engineer working on Cloudbreak

Attila

Re: Pros and cons of using EMR vs Cloudbreak for launching hadoop cluster on AWS

obaid_salikeen — Sun, 30 Oct 2016 01:20:38 GMT

Thanks a lot @Attila Kanto for a detailed response,

Let me ask another cost related question, which is an important factor for making a decision on which technology to use: How would you compare EMR vs Cloudbreak (or Hortonworks Data Cloud) in-terms of cost?

Obaid

Re: Pros and cons of using EMR vs Cloudbreak for launching hadoop cluster on AWS

akanto — Sun, 30 Oct 2016 02:55:34 GMT

Sorry, but I do not have such comparison.

Attila

Re: Pros and cons of using EMR vs Cloudbreak for launching hadoop cluster on AWS

obaid_salikeen — Thu, 03 Nov 2016 20:35:17 GMT

sure, no problem