Hi, I am running hdp2.5 on ubuntu14.04 with hadoop2.7.3.
I am trying to use Docker Executor Container in yarn. I have overrided yarn.nodemanager.container-executor.class with org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor. It seemed that I could restart yarn successfully.
But when i was trying to launch a MR job, I got an error "Container image must not be null". Actually I have set images for containers like this:
-Dmapreduce.map.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.4.1" \ -Dyarn.app.mapreduce.am.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.4.1" \
I knew that hdp2.4.x doesn't support Docker Executor Container from an relevant question.
In short, I have two questions:
1. does hdp2.5.x support Docker Executor Container?
2. if it does, how can I fix the above problem?
Thanks in advance!
I think this seems to be very much work in progress and not officially support yet by Hortonworks though I do see active development on this from our team.
Below are some steps recorded from one of our staff testing this. They themselves had mixed results.
Steps required on each node with a running node manager, before we can start testing -------- a. Ensure that we are NOT running a secure cluster - docker support does not work with secure clusters. b. yum install docker-io - installs docker/dependencies c. service docker start - starts the docker daemon d. docker pull sequenceiq/hadoop-docker:2.5.1 ( ~ 1.5 GB, I think ) - this has to be done in advance before launching containers since its very slow. e. add user yarn to group docker (or whichever user yarn is running as). Add group docker in case it doesn’t exist already. f. modify yarn-site.xml to add yarn.nodemanager.docker-container-executor.exec-name (point to docker binary) and yarn.nodemanager.container-executor.class (to use docker container executor) g. restart node manager
Running a test MR job
I would suggest that maybe this is not mature enough for production but still might be worth while for experimentation.
Thanks very much for the answer in detail!
I think I followed most of the steps except step a. I just reviewed configuration in my ambari server. I confirmed that Admin->Kerberos was disable. But I found some relevant configs in core-site.xml like below.
It seemed that I couldn't delete these configs. So is this cluster in secure mode? If yes, how can I disable security thoroughly?