HDP on Mesos using Marathon, Docker

[#1]

Is it possible to deploy HDP docker container in Mesos using Marathon?

If so, where can i get the docker images from and the Marathon recipes?

If its not possible with the combination above, what are the options to deploy HDP on Mesos? How is it going to be better than running on Yarn?

[#2 · Accepted solution]

@learninghuman

Cloudbreak currently offers a Technical Preview for deployment on Mesos. At a high level, Cloudbreak deployment on Mesos is similar to Cloudbreak implementations on other cloud providers: HDP clusters are provisioned through Ambari with the help of blueprints, and Ambari server and agents run in Docker containers. However, there are some important differences with other cloud providers:

• Cloudbreak expects a "bring your own Mesos" infrastructure, which means that you have to deploy Mesos first and then configure access to the existing Mesos deployment in Cloudbreak.
• The Cloudbreak Mesos integration was designed not to include steps to first build the infrastructure as it does for other cloud environments, such as: creating or reusing the networking layer (virtual networks, subnets, and so on), provisioning new virtual machines in these networks from pre-existing cloud images, and starting docker containers on these VMs (nodes).

From a YARN perspective, there is no difference between Mesos and other public / private clouds. YARN manages the compute capacity provided to it by the IaaS layer.

[#3]

@Tom McCuch Thanks for the clarification. One another related question is that in general what are the advantages that Mesos would bring over Yarn? Especially given the fact that Hortonworks is making efforts to support HDP on Mesos. I mean why care. If HDP on the cloud, its still YARN thats going to be the cluster manager.

[#4]

YARN is designed for Hadoop and is very mature and stable.

Mesos is very new, written in C++, has CPU scheduling.

This presentation is pretty good.

http://www.slideshare.net/mKrishnaKumar1/mesos-vs-yarn-an-overview

[#5]

@learninghuman

To help clarify, all of the data access components within HDP run on YARN. We view Mesos as one of the many alternatives for IaaS within the private cloud space (Openstack, VMware, etc.). Our aim is to support them all and provide our customers both connectivity and portability across them with HDF and HDP.

[#6]

@learninghuman I hope this answered your questions. If so, please remember to accept my answer. Thank you! _Tom