HDP 2.5 Sandbox Docker image cannot be loaded into Docker on Windows 10 Professional

[#1]

Docker fails to load the HDP 2.5 Sandbox image on Windows10, even though Windows 10 now supports Docker. I am using the HDP 2.5 Sandbox image, and the very latest docker versions. I have tried both 1.13 and 1.14 but the problem persists. There is a file in one of the inner tar files inside the image that causes a problem. Actually it is a pair of files with degenerate names that cause the problem:

/usr/share/terminfo/E/eterm

/usr/share/terminfo/E/Eterm

When these files are copied to C:\ProgramData\docker\windowsfilter\..., the second file causes an error when it attempts to overwrite the first. This is because Windows is case-insensitive while linux is not. The full error during loading is this:

re-exec error: exit status 1: output: Failed to OpenForBackup failed in Win32: open \\?\C:\ProgramData\docker\windowsfilter\0688ca7723d72c6b0725e7bc3684a98fb9f565a00c13b552936c46c2d74d6e39\usr\share\terminfo\e\eterm: The file exists. (0x1f) \\?\C:\ProgramData\docker\windowsfilter\0688ca7723d72c6b0725e7bc3684a98fb9f565a00c13b552936c46c2d74d6e39\usr\share\terminfo\e\eterm

I have tried removing the offending files from the archive but then I get a different error about a file not existing. (There is more than one file with this issue in the image.)

I am wondering if anyone else has encountered this problem when attempting to load a linux Docker image on Windows.

[#2]

@Joseph Leiba

This sounds like a Docker on Windows specific issue. The Sandbox image is the same file used for all platforms. The load process works on CentOs/RHEL and Mac OS X. I will test on my Windows 10 computer and report back my findings.

[#3]

Hi Michael, I think you are right. I found the following article that states there are additional steps required to run Linux containers on Windows, and it is experimental. I'm having a little bit of trouble testing this, though, because Docker doesn't work on Windows 10 Pro when Symantec Endpoint Protection is installed.

Run Linux and Windows Containers on Windows 10

By the way, I'm trying to run the Docker image directly on windows because I don't prefer this new type of sandbox that is a Docker image running inside of a VM. I would rather not login two levels deep to get at the Hadoop environment.

[#4 · Accepted solution]

I have had much difficulty getting "Docker for Windows" to run on Windows. (This is the beta release that supposedly supports Linux containers.) At first Docker didn't work at all, even after enabling Hyper-V, Containers, etc, but then I followed the actions performed by one of the users (damtur) on this Docker issue, and now it appears to be running. I have even successfully loaded the HDP 2.5 Sandbox image. My next step is to actually start it and try it out. (We have some experience here as we are already running HDP 2.5 on two clusters.)

(BTW, the official, non-beta release of Docker does not work on Windows 10 Pro if you have antivirus installed, but somehow this beta release does not suffer from the same problem. The main release will fail to load the initial image, as described in this issue. This is a different problem than the one I was having where Docker simply fails to start after installing with the .msi package.)

[#5]

@Joseph Leiba

Thank you for sharing your findings.

[#6]

UPDATE: I can successfully start the container on Windows 10, but you cannot allocate more than 2GB RAM because a bug prevents you from changing any memory or CPU settings. The bug is: Impossible to change memory size #342.

At this point, again, I don't think there is an effective way to run the HDP 2.5 sandbox directly in Docker on Windows.