Docker build on GPU-Host returned a non-zero code: 100

[#1]

I'm trying to create a custom CUDA-capable engine image with the CDSW GPU-Guide
on our airgapped CDSW-Cluster. We have a CDSW Cluster with 1 Master-Node and 2 Woker-Nodes (one of the Worker-Nodes is equipped with NVIDIA GPUs)

I used the following command on the GPU-Host, which is also given in CSDW GPU-Guide, to build the CUDA docker image:

docker build --network host -t <company-registry>/cdsw-cuda:8 . -f cuda.Dockerfile

Beforehand we configured these firewall rules on our GPU-Host:

• http://developer.download.nvidia.com/*
• http://archive.ubuntu.com/ubuntu/*
• http://security.ubuntu.com/ubuntu/*

Unfortunately I get these error messages: (Full output on stdout is attached)

Reading package lists...
W: The repository 'http://archive.cloudera.com/kudu/ubuntu/xenial/amd64/kudu xenial-kudu5 Release' does not have a Release file.
W: The repository 'http://ppa.launchpad.net/deadsnakes/ppa/ubuntu xenial Release' does not have a Release file.
E: Failed to fetch http://archive.cloudera.com/kudu/ubuntu/xenial/amd64/kudu/dists/xenial-kudu5/contrib/source/Sources 403 Forbidden
E: Failed to fetch http://ppa.launchpad.net/deadsnakes/ppa/ubuntu/dists/xenial/main/binary-amd64/Packages 403 Forbidden [IP: 91.189.95.83 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.
The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cuda-cudart-$CUDA_PKG_VERSION && ln -s cuda-10.0 /usr/local/cuda && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

 At this point I have several questions:

• Is it correct to build the docker image on the GPU-Host or should it be done on the CDSW-Master?
• I couldn't find some of the paths on my entire CDSW-Cluster which are given in the cuda.Dockerfile, for instance:

  /etc/apt/sources.list.d/
  OR
  /usr/local/nvidia/lib
  OR
  /var/lib/apt/lists/*

         Is this ok?

• Do airgapped installations need more friewall rules to be configured?

I would be grateful for some kind of solution/feedback on that.

Thanks!

[#2]

I couldn't figure the problem out myself. Can someone please help?

Many thanks!

[#3 · Accepted solution]

After we configured these firewall rules, I was able to build the custom image:

http://archive.cloudera.com/*    PORT: 80
http://ppa.launchpad.net/*       PORT: 80

Hint on docker company-registry: 

I was at the beginning a bit confused about the docker company-registry because we never set up a public docker registry. This is important because when you build and push the docker image, the commands require the company-registry. I could find it simply with the command

docker images

 You will get a list of images registered on your machine. I found out that the company registry in my case was

docker.repository.cloudera.com/cdsw/

 So what I did was just building the docker image with my company-registry. In my case there was no need to push the image to docker. (see CDSW GPU Guide)

docker build --network host -t docker.repository.cloudera.com/cdsw/cdsw-cuda:8 . -f cuda.Dockerfile

After that, the image was built successfully. The image can be listed with the command

docker images

 One should see the newly created docker image in the repository.

For the last step the site admin. has to add this image to CDSW (see CDSW GPU Guide). The Repository:Tag in CDSW would be the same as shown above with the company-registry. In this example it would be

Repository:Tag docker.repository.cloudera.com/cdsw/cdsw-cuda:8

For airgapped installations you need these firewall rules to be able to build the NVIDIA image for CDSW: (all on Port 80)

• http://developer.download.nvidia.com/*
• http://archive.ubuntu.com/ubuntu/*
• http://security.ubuntu.com/ubuntu/*
• http://archive.cloudera.com/*
• http://ppa.launchpad.net/*

Regards.