Update:  I think we could further diagnose the issue. It seems that one of the cdsw pods is constantly crashing. Specifically "nvidia-device-plugin-daemonset-rpkqm" Looking at the logs from one of the crashed Containers it seems k8s is not able to invoke cuda:   container_linux.go:247: starting container process caused "process_linux.go:337: running prestart hook 0 caused \"error running hook: exit status 1, stdout: , stderr: exec command: [/opt/cloudera/parcels/CDSW/nvidia/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig --device=all --utility --pid=807 /var/lib/docker/devicemapper/mnt/f713fe306e9d51cc24017a76d61b1a491533c1f626a3a89737a16f10aefe4015/rootfs]\\nnvidia-container-cli: initialization error: cuda error: os call failed or operation not supported on this os\\n\""   ... View more