Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot enable GPU in Yarn

Cannot enable GPU in Yarn

New Contributor

Following Managing Yarn (from the Cloudera documentation) as well as http://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/UsingGpus.html we've been trying to get Yarn to recognize GPUs as resources but were not able to find "Enable GPU Usage" in Cloudera Manager (step 14 in the documentation).

 

We'd be grateful for any suggestion on what to look for to get our GPUs resources to work.

 

Here's the output of  nvidia-smi on the node where the GPU is installed:

 

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.40       Driver Version: 430.40       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:83:00.0 Off |                    0 |
| N/A   43C    P0    26W /  70W |      0MiB / 15109MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

 

 

1 REPLY 1
Highlighted

Re: Cannot enable GPU in Yarn

New Contributor

P.S. This is what we have in our yarn-site.xml (added in CM through a safety valve, running configuration is in --> /var/run/cloudera-scm-agent/process/12460-yarn-RESOURCEMANAGER-refresh/yarn-site.xml)

<property>
<name>yarn.resource-types</name>
<value>yarn.io/gpu</value>
</property>
<property>
<name>yarn.nodemanager.resource-plugins</name>
<value>yarn.io/gpu</value>
</property>
<property>
<name>yarn.nodemanager.resource-plugins.gpu.path-to-discovery-executables</name>
<value>/usr/bin/nvidia-smi</value>
</property>
<property>
<name>yarn.nodemanager.resource.gpu.allowed-gpu-devices</name>
<value>auto</value>
</property>

 

Don't have an account?
Coming from Hortonworks? Activate your account here