Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why does /etc/hadoop/conf.cloudera.yarn/container-executor.cfg not get used?

Highlighted

Why does /etc/hadoop/conf.cloudera.yarn/container-executor.cfg not get used?

New Contributor

I'm following a collection of sites:

https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/data-operating-system/content/run_docker_conta...

https://blog.cloudera.com/trying-containerized-applications-apache-hadoop-yarn-3-1/
https://hadoop.apache.org/docs/r3.0.0/hadoop-yarn/hadoop-yarn-site/DockerContainers.html
...

so that I can run docker containers using YARN.  However when I update (through Cloudera Manager) the "NodeManager Advanced Configuration Snippet" with the recommended settings:

```
<property><name>yarn.nodemanager.container-executor.class</name><value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value><description>Container executors encapsulate the logic for launching and interacting with containers on a specific Operating System(s).</description></property><property><name>yarn.nodemanager.linux-container-executor.group</name><value>yarn</value><description>The POSIX group of the NodeManager. It should match the setting in "container-executor.cfg". This configuration is required for validating the secure access of the container-executor binary.</description></property><property><name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name><value>false</value><description>Whether all applications should be run as the NodeManager process' owner. When false, applications are launched instead as the application owner.</description></property><property><name>yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user</name><value>nobody</value><description>The user that containers will run as in nonsecure mode. Using root user is not secure.</description></property><property><name>yarn.nodemanager.runtime.linux.allowed-runtimes</name><value>default,docker</value><description>Comma separated list of runtimes that are allowed when using LinuxContainerExecutor.</description></property><property><name>yarn.nodemanager.runtime.linux.docker.capabilities</name><value>CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE</value><description>This configuration setting determines the capabilities assigned to docker containers when they are launched. While these may not be case-sensitive from a docker perspective, it is best to keep these uppercase. To run without any capabilities, set this value to "none" or "NONE". Admins must update this list based on the security requirements of their workloads.</description></property><property><name>yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed</name><value>false</value><description>This configuration setting determines if privileged docker containers are allowed on this cluster. The submitting user must be part of the privileged container acl and must be part of the docker group or have sudo access to the docker command to be able to use a privileged container. Use with extreme care.</description></property><property><name>yarn.nodemanager.runtime.linux.docker.privileged-containers.acl</name><value></value><description>This configuration setting determines the submitting users who are allowed to run privileged docker containers on this cluster. The submitting user must also be part of the docker group or have sudo access to the docker command. No users are allowed by default. Use with extreme care.</description></property><property><name>yarn.nodemanager.runtime.linux.docker.allowed-container-networks</name><value>host,bridge</value><description>The set of Docker networks allowed when launching containers.</description></property><property><name>yarn.nodemanager.runtime.linux.docker.default-container-network</name><value>host</value><description>The Docker network used when launching containers when no network is specified in the request. This network must be one of the (configurable) set of allowed container networks. The default is host, which may not be appropriate for multiple containers on a single node when they use the same port, use bridge in that case. See docker networking documentation for more.</description></property>
```

And I create the /etc/hadoop/conf.cloudera.yarn/container-executor.cfg file (on each NodeManager):

```

yarn.nodemanager.linux-container-executor.group=yarn

banned.users=hdfs,yarn,mapred,bin

min.user.id=50

                                

[docker]

module.enabled=true

docker.binary=/usr/bin/docker

docker.allowed.capabilities=CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,

                            SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,

                            SYS_CHROOT,KILL,AUDIT_WRITE,DAC_READ_SEARCH,

                            SYS_PTRACE,SYS_ADMIN

docker.allowed.networks=bridge,host,none

docker.allowed.ro-mounts=/sys/fs/cgroup

docker.privileged-containers.enabled=false

 

docker.trusted.registries=local,centos,hortonworks
```

Then I restart YARN service but the container-executor.cfg files are of different sizes:

```

[root@node1 hadoop-conf]# ls -al /etc/hadoop/conf.cloudera.yarn/container-executor.cfg

-rw-r--r-- 1 root hadoop 645 Sep 20 16:14 /etc/hadoop/conf.cloudera.yarn/container-executor.cfg

[root@node1 hadoop-conf]# ls -al /run/cloudera-scm-agent/process/936-yarn-NODEMANAGER/container-executor.cfg

-r-------- 1 root hadoop 156 Sep 23 09:07 /run/cloudera-scm-agent/process/936-yarn-NODEMANAGER/container-executor.cfg
```

The contents of the files are different also so it seems like when restarting YARN it is ignoring the 

/etc/hadoop/conf.cloudera.yarn/container-executor.cfg and generating a default 

/run/cloudera-scm-agent/process/936-yarn-NODEMANAGER/container-executor.cfg.  

 

Lastly, when I try to run a docker container using YARN:

```

sudo -u nobody yarn --debug jar

/opt/cloudera/parcels/CDH/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar

\

-shell_env YARN_CONTAINER_RUNTIME_TYPE="docker" \

-shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE="centos:latest" \

-shell_command "sleep 10" \

-jar

/opt/cloudera/parcels/CDH/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar

\

-num_containers 1

```

 

I'm getting this in the logs:

```

19/09/23 13:22:23 INFO distributedshell.ApplicationMaster: appattempt_1569262298521_0001_000001 got container status for containerID=container_1569262298521_0001_01_000002, state=COMPLETE, exitStatus=32, diagnostics=[2019-09-23 13:22:18.620]Exception from container-launch.
Container id: container_1569262298521_0001_01_000002
Exit code: 32
Exception message: Feature disabled: docker

Shell output: main : command provided 4
main : run as user is nobody
main : requested yarn user is nobody


[2019-09-23 13:22:18.670]Container exited with a non-zero exit code 32. 
[2019-09-23 13:22:18.670]Container exited with a non-zero exit code 32. 

19/09/23 13:22:24 INFO distributedshell.ApplicationMaster: Application completed. Stopping running containers
19/09/23 13:22:24 INFO distributedshell.ApplicationMaster: Application completed. Signalling finish to RM
19/09/23 13:22:24 INFO distributedshell.ApplicationMaster: Diagnostics., total=1, completed=1, allocated=1, failed=1
19/09/23 13:22:24 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
19/09/23 13:22:24 INFO distributedshell.ApplicationMaster: Application Master failed. exiting

```

 

And when running "docker ps -a" and "docker images" I can tell no image was pulled and ran.


Thank you to all that can help!

Relatable Details:

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here