Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

avatar
Explorer

The hadoop documentation states that DCE does not support a cluster with secure mode (Kerberos): https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html

Are people working on this? Is there a way around this limitation?

1 ACCEPTED SOLUTION

avatar
New Contributor

@rpul DCE has significant drawbacks. YARN-3611 is the umbrella JIRA for a new mechanism of docker integration on YARN. This mechanism can be used on a kerberized cluster (it does add some requirements for the images being used - the user account needs to be available in the image as well, for example).

View solution in original post

6 REPLIES 6

avatar

This feature is tracked in YARN-2477 (DockerContainerExecutor must support secure mode). There is currently no fix version specified. Docker container support on YARN is still very new, so you might want to follow the umbrella JIRA, (YARN-2466) to gain an idea of when additional features might become available.

avatar
Explorer

Thanks for the JIRA's Alex!

The changes to them are over 1 year old. Do you think DCE will be supported on a kerberized cluster?

DCE seems like the way to go for running isolated jobs in production while retaining data locality. Any thoughts on why these JIRA's aren't picked up?

avatar
Explorer

@sseethana: You seem to have been working a lot on YARN-3611. Can you give an update on the current efforts and status of running Docker on YARN?

avatar
New Contributor

@rpul DCE has significant drawbacks. YARN-3611 is the umbrella JIRA for a new mechanism of docker integration on YARN. This mechanism can be used on a kerberized cluster (it does add some requirements for the images being used - the user account needs to be available in the image as well, for example).

avatar
Explorer

@sseethana: Thanks for the info! Is any of this functionality already available in a released Hadoop distribution of Apache or Hortonworks? If so, is there any documentation or a getting started guide?

avatar
New Contributor

@rpul Many of the features are going to be in the 2.8 (community) release. There is an open ticket for (YARN-5258) to document these features - hopefully we can address this soon.