Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Using Docker containers as IaaS for Hortonworks HDP?

Using Docker containers as IaaS for Hortonworks HDP?

New Contributor

I am new to this whole containerization methodology and am wondering if my intended cluster architecture is a misconception. We have a Openshift cluster, which runs Docker containers.

What I want to do is the following:

  1. Set up 10 somewhat docker containers which run a naked RHEL image
  2. Set up Ambari Server on one of the containers, which will then successively begin to provision the other nodes in the cluster
  3. Keep these containers alive 24/7 and highly available (spawn new containers when some containers crash)

Unfortunately we can not use Hortonworks Cloudbreak with OpenShift as it is not compatible at the moment. So this is my workaround solution. What I obtain is a containerized cluster that should just act like typical (virtual) servers - is this assumption correct?

What are the pitfalls of a solution like this? It looks good to me on the paper and I could not think of major limitations right now. We have dedicated storage nodes that can persist data that is written in the containers.

3 REPLIES 3

Re: Using Docker containers as IaaS for Hortonworks HDP?

It depends on your setup, but maybe Ambari is not necessary at all.

You can find more information about the typical problems to solve here and some working docker containers here and here.

Hope this helps!

Highlighted

Re: Using Docker containers as IaaS for Hortonworks HDP?

New Contributor

I want to ask more specifically:

So it is not a problem to start with a naked RHEL docker container and successively add/install new packages to it? I.e. add Ambari Agent, HDFS, Hive, etc. to one single the container? If I understood correctly, it looks like this is what Hortonworks Cloudbreak is doing.

This would make provisioning and managing our cluster much easier. We would begin to use separate docker containers after our solution with the "big" docker container works.

Re: Using Docker containers as IaaS for Hortonworks HDP?

@Jody H

It is not a problem to have all the services in a single container, it might be a Docker anti-pattern, but it should work, you can go that way.

Cloudbreak has not been creating HDP clusters in Docker containers for ~2 years now, before then CB deployed into containers, but there were some stability and enterprise supportability issues with that approach back then, which might have been resolved since then.

Hope this helps!