- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Can I run DCE (Docker Container Executor) on Yarn with Kerberos?
- Labels:
-
Apache YARN
-
Docker
Created 06-10-2016 01:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The hadoop documentation states that DCE does not support a cluster with secure mode (Kerberos): https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html
Are people working on this? Is there a way around this limitation?
Created 06-15-2016 09:49 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@rpul DCE has significant drawbacks. YARN-3611 is the umbrella JIRA for a new mechanism of docker integration on YARN. This mechanism can be used on a kerberized cluster (it does add some requirements for the images being used - the user account needs to be available in the image as well, for example).
Created 06-10-2016 06:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This feature is tracked in YARN-2477 (DockerContainerExecutor must support secure mode). There is currently no fix version specified. Docker container support on YARN is still very new, so you might want to follow the umbrella JIRA, (YARN-2466) to gain an idea of when additional features might become available.
Created 06-13-2016 08:44 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the JIRA's Alex!
The changes to them are over 1 year old. Do you think DCE will be supported on a kerberized cluster?
DCE seems like the way to go for running isolated jobs in production while retaining data locality. Any thoughts on why these JIRA's aren't picked up?
Created 06-15-2016 12:45 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@sseethana: You seem to have been working a lot on YARN-3611. Can you give an update on the current efforts and status of running Docker on YARN?
Created 06-15-2016 09:49 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@rpul DCE has significant drawbacks. YARN-3611 is the umbrella JIRA for a new mechanism of docker integration on YARN. This mechanism can be used on a kerberized cluster (it does add some requirements for the images being used - the user account needs to be available in the image as well, for example).
Created 06-17-2016 07:50 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@sseethana: Thanks for the info! Is any of this functionality already available in a released Hadoop distribution of Apache or Hortonworks? If so, is there any documentation or a getting started guide?
Created 06-17-2016 08:06 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@rpul Many of the features are going to be in the 2.8 (community) release. There is an open ticket for (YARN-5258) to document these features - hopefully we can address this soon.