Not able to run docker container on yarn even after configuring all the recommendations for running Docker containers on YARN

[#1]

Hi,

I followed all the recommendatiions and pre-requisite to execute docker containers on Yarn but failed to launch container.

I have used single node HDP 3.0 non-kerberized cluster.

Link used for recommendations:

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/data-operating-system/managing-data-operati...

Ran mapreduce sample job to launch docker container but got the below error:


18/08/16 15:34:59 INFO distributedshell.Client: Got application report from ASM for, appId=2, clientToAMToken=null, appDiagnostics=Application Failure: desired = 1, completed = 1, allocated = 1, failed = 1, diagnostics = [2018-08-16 15:34:53.201]Exception from container-launch.
Container id: container_e09_1534445826689_0002_01_000002
Exit code: 127
Exception message: Launch container failed
Shell error output: image: library/centos:latest is not trusted.
Disable mount volume for untrusted image
image: library/centos:latest is not trusted.
Disable mount volume for untrusted image
image: library/centos:latest is not trusted.
Disable cap-add for untrusted image
Docker capability disabled for untrusted image

Shell output: main : command provided 4
main : run as user is nobody
main : requested yarn user is root
bdaf395fd9915b71dceeedb996c4d3af092e90c651628e83b201200634b3be23
Creating script paths...
Creating local dirs...
Getting exit code file...
Changing effective user to root...
Inspecting docker container...
Docker inspect command: /usr/bin/docker inspect --format {{.State.Pid}} container_e09_1534445826689_0002_01_000002
pid from docker inspect: 0
Obtaining the exit code...
Docker inspect command: /usr/bin/docker inspect --format {{.State.ExitCode}} container_e09_1534445826689_0002_01_000002
Exit code from docker inspect: 127
Wrote the exit code 127 to /hadoop/yarn/local/nmPrivate/application_1534445826689_0002/container_e09_1534445826689_0002_01_000002/container_e09_1534445826689_0002_01_000002.pid.exitcode

[2018-08-16 15:34:53.232]Container exited with a non-zero exit code 127. Last 4096 bytes of stderr.txt :

[2018-08-16 15:34:53.233]Container exited with a non-zero exit code 127. Last 4096 bytes of stderr.txt :

Please advice on the above issue.

[#2]

What have you configured as "docker.trusted.registries"? You will need to configure it according to your setup. In a typical setup you create your own docker registry and provide it as trusted. If you don't have already one, you can use this image to run a registry:

https://hub.docker.com/_/registry/

[#3]

Thanks! Harald for your input. It works finally after setting up local registry as mentioned in the provided link.

[#4]

Glad it is working. It would be great if you could then also accept the answer. This will indicate other viewers your issue is solved, and increases your and mine reputation points.

[#5]

I am getting the same error while running docker container, could you please mention how to setup local registry.. I am new to docker

[#6]

It's described at the link, but its just a few steps (actually for a test setup):

docker pull registry
docker run -d -p 5000:5000 --restart always --name registry registry:2

Now on the machine where you executed the above commands, a docker registry is available at port 5000.

The parameter have this meaning:
-d: Run container in background and print container ID
-p: Publish a container’s port(s) to the host
--restart: Restart policy to apply when a container exits
--name: Assign a name to the container

For further options refer to: https://docs.docker.com/engine/reference/commandline/run/

For informations on how to set it up for real use (not testing nor demonstration):

https://docs.docker.com/registry/deploying/

The last link provides you with important information like how to set up keys and use it in an orchestrated environment.

[#7]

I am also getting this same error. I configured local registry. What should be the value of docker.trusted.registries? I tried docker.trusted.registries=local,centos,hortonworks but still getting the same error

[#8 · Accepted solution]

We got it working by adding a tag to centos image with below commands:

docker tag centos local/centos

Here is the modified distributed shell command to run:

yarn jar $DJAR -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=local/centos -shell_command "sleep 120"-jar $DJAR -num_containers 1

Note - For multi-node cluster, you will have to run docker tag command on every node manager as root user.

Please also make sure that you have added "local" registry as trusted registry in yarn configurations.

Hope this helps!

Special thanks to @rmaruthiyodan