Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

[#1]

The hadoop documentation states that DCE does not support a cluster with secure mode (Kerberos): https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html

Are people working on this? Is there a way around this limitation?

[#2]

This feature is tracked in YARN-2477 (DockerContainerExecutor must support secure mode). There is currently no fix version specified. Docker container support on YARN is still very new, so you might want to follow the umbrella JIRA, (YARN-2466) to gain an idea of when additional features might become available.

[#3]

Thanks for the JIRA's Alex!

The changes to them are over 1 year old. Do you think DCE will be supported on a kerberized cluster?

DCE seems like the way to go for running isolated jobs in production while retaining data locality. Any thoughts on why these JIRA's aren't picked up?

[#4]

@sseethana: You seem to have been working a lot on YARN-3611. Can you give an update on the current efforts and status of running Docker on YARN?

[#5 · Accepted solution]

@rpul DCE has significant drawbacks. YARN-3611 is the umbrella JIRA for a new mechanism of docker integration on YARN. This mechanism can be used on a kerberized cluster (it does add some requirements for the images being used - the user account needs to be available in the image as well, for example).

[#6]

@sseethana: Thanks for the info! Is any of this functionality already available in a released Hadoop distribution of Apache or Hortonworks? If so, is there any documentation or a getting started guide?

[#7]

@rpul Many of the features are going to be in the 2.8 (community) release. There is an open ticket for (YARN-5258) to document these features - hopefully we can address this soon.