question Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos? in Archives of Support Questions (Read Only)

Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

rpul — Fri, 10 Jun 2016 20:18:58 GMT

The hadoop documentation states that DCE does not support a cluster with secure mode (Kerberos): https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html

Are people working on this? Is there a way around this limitation?

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

amiller — Sat, 11 Jun 2016 01:18:43 GMT

This feature is tracked in YARN-2477 (DockerContainerExecutor must support secure mode). There is currently no fix version specified. Docker container support on YARN is still very new, so you might want to follow the umbrella JIRA, (YARN-2466) to gain an idea of when additional features might become available.

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

rpul — Mon, 13 Jun 2016 15:44:45 GMT

Thanks for the JIRA's Alex!

The changes to them are over 1 year old. Do you think DCE will be supported on a kerberized cluster?

DCE seems like the way to go for running isolated jobs in production while retaining data locality. Any thoughts on why these JIRA's aren't picked up?

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

rpul — Wed, 15 Jun 2016 19:45:20 GMT

@sseethana: You seem to have been working a lot on YARN-3611. Can you give an update on the current efforts and status of running Docker on YARN?

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

sseethana — Thu, 16 Jun 2016 04:49:11 GMT

@rpul DCE has significant drawbacks. YARN-3611 is the umbrella JIRA for a new mechanism of docker integration on YARN. This mechanism can be used on a kerberized cluster (it does add some requirements for the images being used - the user account needs to be available in the image as well, for example).

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

rpul — Fri, 17 Jun 2016 14:50:26 GMT

@sseethana: Thanks for the info! Is any of this functionality already available in a released Hadoop distribution of Apache or Hortonworks? If so, is there any documentation or a getting started guide?

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

sseethana — Sat, 18 Jun 2016 03:06:36 GMT

@rpul Many of the features are going to be in the 2.8 (community) release. There is an open ticket for (YARN-5258) to document these features - hopefully we can address this soon.