question Re: Can Yarn on CDH 5.7 launch (docker) containers running servers? in Archives of Support Questions (Read Only)

Can Yarn on CDH 5.7 launch (docker) containers running servers?

BillM. — Fri, 16 Sep 2022 11:14:59 GMT

Hello:

We have some CDH 5.7 clusters, and we want to deploy servers in containers to the nodes (probably via yarn).

Informally, we have some workflows that contact remote servers to do a fairly complex computation, and are interested in using cluster nodes to host these servers (to promote more elastic use of our hardware). One approach we are entertaining is to attempt to package these servers in containers (docker is the current candidate), and we believe that some work has been done to launch docker containers via Yarn (e.g. https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html). My questions are:

Is it possible to deploy user configured containers (e.g. docker) to deploy servers via Yarn?
If so, would docker be a suitable container technology for this use case? If not docker, what would be better?
If we were to use docker containers for this use case, would all jobs in the system need to use docker containers? How does that affect the build process and (oozie based) deployment?

Thanks:

Bill

Re: Can Yarn on CDH 5.7 launch (docker) containers running servers?

BillM. — Thu, 16 Mar 2017 19:27:23 GMT

Hello:

My apologoies, there is an error in the subject, it should have read CDH 5.7 not CDH 7.5. I'm trying to fix it.

Also, if a newer version of CDH has support (or if it is on the road map) I'd be interested in hearing about it.

With best regards:

Bill M.

Re: Can Yarn on CDH 5.7 launch (docker) containers running servers?

Dat Pham — Tue, 28 Mar 2017 23:37:22 GMT

There is no plan to support Docker with CDH at the moment.

Re: Can Yarn on CDH 5.7 launch (docker) containers running servers?

jamshid — Mon, 03 Apr 2017 22:29:58 GMT

So this is no longer supported? Why -- technical or business reasons? Anyway, maybe you could update it to make that clear?

https://hub.docker.com/r/cloudera/quickstart/

How about clusterdock, Cloudera blogged about it last year:

http://blog.cloudera.com/blog/2016/08/multi-node-clusters-with-cloudera-quickstart-for-docker/

Will this image be supported as an easy Production-like environment?

https://hub.docker.com/r/cloudera/clusterdock/

Thanks,

Jamshid