Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

Solved Go to solution
Highlighted

Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

New Contributor

The hadoop documentation states that DCE does not support a cluster with secure mode (Kerberos): https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html

Are people working on this? Is there a way around this limitation?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

New Contributor

@rpul DCE has significant drawbacks. YARN-3611 is the umbrella JIRA for a new mechanism of docker integration on YARN. This mechanism can be used on a kerberized cluster (it does add some requirements for the images being used - the user account needs to be available in the image as well, for example).

View solution in original post

6 REPLIES 6
Highlighted

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

This feature is tracked in YARN-2477 (DockerContainerExecutor must support secure mode). There is currently no fix version specified. Docker container support on YARN is still very new, so you might want to follow the umbrella JIRA, (YARN-2466) to gain an idea of when additional features might become available.

Highlighted

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

New Contributor

Thanks for the JIRA's Alex!

The changes to them are over 1 year old. Do you think DCE will be supported on a kerberized cluster?

DCE seems like the way to go for running isolated jobs in production while retaining data locality. Any thoughts on why these JIRA's aren't picked up?

Highlighted

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

New Contributor

@sseethana: You seem to have been working a lot on YARN-3611. Can you give an update on the current efforts and status of running Docker on YARN?

Highlighted

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

New Contributor

@rpul DCE has significant drawbacks. YARN-3611 is the umbrella JIRA for a new mechanism of docker integration on YARN. This mechanism can be used on a kerberized cluster (it does add some requirements for the images being used - the user account needs to be available in the image as well, for example).

View solution in original post

Highlighted

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

New Contributor

@sseethana: Thanks for the info! Is any of this functionality already available in a released Hadoop distribution of Apache or Hortonworks? If so, is there any documentation or a getting started guide?

Re: Can I run DCE (Docker Container Executor) on Yarn with Kerberos?

New Contributor

@rpul Many of the features are going to be in the 2.8 (community) release. There is an open ticket for (YARN-5258) to document these features - hopefully we can address this soon.

Don't have an account?
Coming from Hortonworks? Activate your account here