Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP on Mesos using Marathon, Docker

SOLVED Go to solution
Highlighted

HDP on Mesos using Marathon, Docker

Contributor

Is it possible to deploy HDP docker container in Mesos using Marathon?

If so, where can i get the docker images from and the Marathon recipes?

If its not possible with the combination above, what are the options to deploy HDP on Mesos? How is it going to be better than running on Yarn?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HDP on Mesos using Marathon, Docker

@learninghuman

Cloudbreak currently offers a Technical Preview for deployment on Mesos. At a high level, Cloudbreak deployment on Mesos is similar to Cloudbreak implementations on other cloud providers: HDP clusters are provisioned through Ambari with the help of blueprints, and Ambari server and agents run in Docker containers. However, there are some important differences with other cloud providers:

  • Cloudbreak expects a "bring your own Mesos" infrastructure, which means that you have to deploy Mesos first and then configure access to the existing Mesos deployment in Cloudbreak.
  • The Cloudbreak Mesos integration was designed not to include steps to first build the infrastructure as it does for other cloud environments, such as: creating or reusing the networking layer (virtual networks, subnets, and so on), provisioning new virtual machines in these networks from pre-existing cloud images, and starting docker containers on these VMs (nodes).

From a YARN perspective, there is no difference between Mesos and other public / private clouds. YARN manages the compute capacity provided to it by the IaaS layer.

5 REPLIES 5

Re: HDP on Mesos using Marathon, Docker

@learninghuman

Cloudbreak currently offers a Technical Preview for deployment on Mesos. At a high level, Cloudbreak deployment on Mesos is similar to Cloudbreak implementations on other cloud providers: HDP clusters are provisioned through Ambari with the help of blueprints, and Ambari server and agents run in Docker containers. However, there are some important differences with other cloud providers:

  • Cloudbreak expects a "bring your own Mesos" infrastructure, which means that you have to deploy Mesos first and then configure access to the existing Mesos deployment in Cloudbreak.
  • The Cloudbreak Mesos integration was designed not to include steps to first build the infrastructure as it does for other cloud environments, such as: creating or reusing the networking layer (virtual networks, subnets, and so on), provisioning new virtual machines in these networks from pre-existing cloud images, and starting docker containers on these VMs (nodes).

From a YARN perspective, there is no difference between Mesos and other public / private clouds. YARN manages the compute capacity provided to it by the IaaS layer.

Re: HDP on Mesos using Marathon, Docker

Contributor

@Tom McCuch Thanks for the clarification. One another related question is that in general what are the advantages that Mesos would bring over Yarn? Especially given the fact that Hortonworks is making efforts to support HDP on Mesos. I mean why care. If HDP on the cloud, its still YARN thats going to be the cluster manager.

Re: HDP on Mesos using Marathon, Docker

Super Guru

YARN is designed for Hadoop and is very mature and stable.

Mesos is very new, written in C++, has CPU scheduling.

This presentation is pretty good.

http://www.slideshare.net/mKrishnaKumar1/mesos-vs-yarn-an-overview

Re: HDP on Mesos using Marathon, Docker

@learninghuman

To help clarify, all of the data access components within HDP run on YARN. We view Mesos as one of the many alternatives for IaaS within the private cloud space (Openstack, VMware, etc.). Our aim is to support them all and provide our customers both connectivity and portability across them with HDF and HDP.

Re: HDP on Mesos using Marathon, Docker

@learninghuman I hope this answered your questions. If so, please remember to accept my answer. Thank you! _Tom