Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)

Objective

Many people work exclusively from a laptop where storage space is typically limited to 500GB of space or less. Over time, you may find your available storage space has become a regular concern. It's not uncommon to use an external hard drive to augment available storage space.

The current version of Docker for Mac (1.12.x) does not provide a configuration setting which allows users to change the location where the Docker virtual machine image is located. This means the image, which can grow up to 64GB in size by default, is located on your laptop's primary hard drive.

With the HDP 2.5 version of the Hortonworks sandbox available as a native Docker image, you may find a desire to have more room available to Docker. This tutorial will guide you through the process of moving your Docker virtual machine image to a different location, an external drive in this case. This will free up to 64GB of space on your primary laptop hard drive and let you expand the size of the Docker image file later. This tutorial is the first in a two part series.

Prerequisites

Scope

  • Mac OS X 10.11.6 (El Capitan)
  • Docker for Mac 1.12.1
  • HDP 2.5 Docker Sandbox

Steps

Stop Docker for Mac

Before we can make any changes to the Docker virtual machine image, we need to stop Docker for Mac. There should be a Docker for Mac icon in the menu bar. You should see something similar to this:

9219-docker-for-mac-menu.png

You can also check via the command line via the ps -ef | grep -i com.docker. You should see something similar to this:

ps -ef | grep -i com.docker
0   123     1   0  8:45AM ??         0:00.01 /Library/PrivilegedHelperTools/com.docker.vmnetd
502   967   876   0  8:46AM ??         0:00.08 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux -watchdog fd:0
502   969   967   0  8:46AM ??         0:00.04 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux -watchdog fd:0
502   971   967   0  8:46AM ??         0:07.96 com.docker.db --url fd:3 --git /Users/myoung/Library/Containers/com.docker.docker/Data/database
502   975   967   0  8:46AM ??         0:03.40 com.docker.osx.hyperkit.linux
502   977   975   0  8:46AM ??         0:00.03 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux
502 12807   967   0  9:17PM ??         0:00.08 com.docker.osxfs --address fd:3 --connect /Users/myoung/Library/Containers/com.docker.docker/Data/@connect --control fd:4 --volume-control fd:5 --database /Users/myoung/Library/Containers/com.docker.docker/Data/s40
502 12810   967   0  9:17PM ??         0:00.12 com.docker.slirp --db /Users/myoung/Library/Containers/com.docker.docker/Data/s40 --ethernet fd:3 --port fd:4 --vsock-path /Users/myoung/Library/Containers/com.docker.docker/Data/@connect --max-connections 900
502 12811   967   0  9:17PM ??         0:00.19 com.docker.driver.amd64-linux -db /Users/myoung/Library/Containers/com.docker.docker/Data/s40 -osxfs-volume /Users/myoung/Library/Containers/com.docker.docker/Data/s30 -slirp /Users/myoung/Library/Containers/com.docker.docker/Data/s50 -vmnet /var/tmp/com.docker.vmnetd.socket -port /Users/myoung/Library/Containers/com.docker.docker/Data/s51 -vsock /Users/myoung/Library/Containers/com.docker.docker/Data -docker /Users/myoung/Library/Containers/com.docker.docker/Data/s60 -addr fd:3 -debug
502 12812 12811   0  9:17PM ??         0:00.02 /Applications/Docker.app/Contents/MacOS/com.docker.driver.amd64-linux -db /Users/myoung/Library/Containers/com.docker.docker/Data/s40 -osxfs-volume /Users/myoung/Library/Containers/com.docker.docker/Data/s30 -slirp /Users/myoung/Library/Containers/com.docker.docker/Data/s50 -vmnet /var/tmp/com.docker.vmnetd.socket -port /Users/myoung/Library/Containers/com.docker.docker/Data/s51 -vsock /Users/myoung/Library/Containers/com.docker.docker/Data -docker /Users/myoung/Library/Containers/com.docker.docker/Data/s60 -addr fd:3 -debug
502 12814 12811   0  9:17PM ??         0:16.48 /Applications/Docker.app/Contents/MacOS/com.docker.hyperkit -A -m 12G -c 6 -u -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-vpnkit,uuid=1f629fed-1ef6-4f34-8fce-753347e3b941,path=/Users/myoung/Library/Containers/com.docker.docker/Data/s50,macfile=/Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.0 -s 3,virtio-blk,file:///Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2,format=qcow -s 4,virtio-9p,path=/Users/myoung/Library/Containers/com.docker.docker/Data/s40,tag=db -s 5,virtio-rnd -s 6,virtio-9p,path=/Users/myoung/Library/Containers/com.docker.docker/Data/s51,tag=port -s 7,virtio-sock,guest_cid=3,path=/Users/myoung/Library/Containers/com.docker.docker/Data,guest_forwards=2376;1525 -l com1,autopty=/Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty,log=/Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/console-ring -f kexec,/Applications/Docker.app/Contents/Resources/moby/vmlinuz64,/Applications/Docker.app/Contents/Resources/moby/initrd.img,earlyprintk=serial console=ttyS0 com.docker.driver="com.docker.driver.amd64-linux", com.docker.database="com.docker.driver.amd64-linux" ntp=gateway mobyplatform=mac -F /Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/hypervisor.pid
502 13790   876   0  9:52PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.frontend {"action":"vmstateevent","args":{"vmstate":"running"}}
502 13791 13790   0  9:52PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.frontend {"action":"vmstateevent","args":{"vmstate":"running"}}
502 13793 13146   0  9:52PM ttys000    0:00.00 grep -i com.docker

Now we are going to stop Docker for Mac. Before shutting down Docker, make sure all of your containers have been stopped. Using the menu shown above, click on the Quit Docker menu option. This will stop Docker for Mac. You should notice the Docker for Mac icon is no longer visible.

Now let's confirm the Docker processes we saw before are no longer running:

ps -ef | grep -i com.docker
0   123     1   0  8:45AM ??         0:00.01 /Library/PrivilegedHelperTools/com.docker.vmnetd
502 13815 13146   0  9:54PM ttys000    0:00.00 grep -i com.docker

NOTE: It may take a minute or two before Docker completely shuts down.

Backup Docker virtual machine image

Before we make any changes to the Docker virtual machine image, we should back it up. This will temporarily use more space on your laptop hard drive. Make sure you have enough room to hold two copies of the data. As mentioned before, the Docker image can be up to 64GB by default. Let's check the current size of our image using du -sh. The Docker image file is located at ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ by default.

du -sh ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/
 64G    /Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/

In my case, my image size is 64GB. You need to be sure you have room for 2 copies of the com.docker.driver.amd64-linux directory. Now we'll make a copy of our image:

cd ~/Library/Containers/com.docker.docker/Data/
cp -r com.docker.driver.amd64-linux com.docker.driver.amd64-linux.backup

This copy serves as our backup of the image.

Copy Docker virtual machine image to external drive

Now we can make a copy of our image on our external hard drive. I have a 1TB SSD mounted at /Volumes/Samsung. I am going to store my Docker virtual machine image in /Volumes/Samsung/Docker/image. You should store the image in a location that makes sense for you.

cp -r com.docker.driver.amd64-linux /Volumes/Samsung/Docker/image/

This process will take a few minutes. It will take longer if you are note using an SSD. Let's confirm the directory now exists on the external hard drive.

ls -la /Volumes/Samsung/Docker/image/
total 0
drwxr-xr-x   3 myoung  staff  102 Nov  3 17:08 .
drwxr-xr-x  11 myoung  staff  374 Nov  3 17:03 ..
drwxr-xr-x@ 11 myoung  staff  374 Nov  7 21:53 com.docker.driver.amd64-linux

You can also check the size:

du -sh /Volumes/Samsung/Docker/image/
 64G    /Volumes/Samsung/Docker/image/

Now that we have a copy of the Docker image on the external hard drive, we will use a symbolic link from the image directory on the laptop hard drive to image directory on the external hard drive. Before creating the link, we need to remove the current image directory on our laptop hard drive

rm -rf ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux

Now let's create the symbolic link. We will use the ln -s command. The syntax for ln is ln -s <target> <source>. In this case, target is the location on the external drive and source is the location on the internal drive.

ln -s /Volumes/Samsung/Docker/image/com.docker.driver.amd64-linux ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux

We can confirm the link was created:

ls -la ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
lrwxr-xr-x  1 myoung  staff  59 Nov  3 17:05 /Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux -> /Volumes/Samsung/Docker/image/com.docker.driver.amd64-linux

Restart Docker for Mac

Now we can restart Docker for Mac. This is done by running the application from the Applications folder in the Finder. You should see something similar to this:

9221-docker-app.png

Double-click on the Docker application to start it. You should notice the Docker for Mac icon is now back in the main menu bar. You can also check via ps -ef | grep -i com.docker. You should see something similar to this:

ps -ef | grep -i com.docker
0   123     1   0  8:45AM ??         0:00.01 /Library/PrivilegedHelperTools/com.docker.vmnetd
502 14476 14465   0 10:42PM ??         0:00.03 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux -watchdog fd:0
502 14479 14476   0 10:42PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux -watchdog fd:0
502 14480 14476   0 10:42PM ??         0:00.29 com.docker.db --url fd:3 --git /Users/myoung/Library/Containers/com.docker.docker/Data/database
502 14481 14476   0 10:42PM ??         0:00.08 com.docker.osxfs --address fd:3 --connect /Users/myoung/Library/Containers/com.docker.docker/Data/@connect --control fd:4 --volume-control fd:5 --database /Users/myoung/Library/Containers/com.docker.docker/Data/s40
502 14482 14476   0 10:42PM ??         0:00.04 com.docker.slirp --db /Users/myoung/Library/Containers/com.docker.docker/Data/s40 --ethernet fd:3 --port fd:4 --vsock-path /Users/myoung/Library/Containers/com.docker.docker/Data/@connect --max-connections 900
502 14483 14476   0 10:42PM ??         0:00.05 com.docker.osx.hyperkit.linux
502 14484 14476   0 10:42PM ??         0:00.08 com.docker.driver.amd64-linux -db /Users/myoung/Library/Containers/com.docker.docker/Data/s40 -osxfs-volume /Users/myoung/Library/Containers/com.docker.docker/Data/s30 -slirp /Users/myoung/Library/Containers/com.docker.docker/Data/s50 -vmnet /var/tmp/com.docker.vmnetd.socket -port /Users/myoung/Library/Containers/com.docker.docker/Data/s51 -vsock /Users/myoung/Library/Containers/com.docker.docker/Data -docker /Users/myoung/Library/Containers/com.docker.docker/Data/s60 -addr fd:3 -debug
502 14485 14483   0 10:42PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.osx.hyperkit.linux
502 14486 14484   0 10:42PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.driver.amd64-linux -db /Users/myoung/Library/Containers/com.docker.docker/Data/s40 -osxfs-volume /Users/myoung/Library/Containers/com.docker.docker/Data/s30 -slirp /Users/myoung/Library/Containers/com.docker.docker/Data/s50 -vmnet /var/tmp/com.docker.vmnetd.socket -port /Users/myoung/Library/Containers/com.docker.docker/Data/s51 -vsock /Users/myoung/Library/Containers/com.docker.docker/Data -docker /Users/myoung/Library/Containers/com.docker.docker/Data/s60 -addr fd:3 -debug
502 14488 14484   0 10:42PM ??         0:07.90 /Applications/Docker.app/Contents/MacOS/com.docker.hyperkit -A -m 12G -c 6 -u -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-vpnkit,uuid=1f629fed-1ef6-4f34-8fce-753347e3b941,path=/Users/myoung/Library/Containers/com.docker.docker/Data/s50,macfile=/Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.0 -s 3,virtio-blk,file:///Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2,format=qcow -s 4,virtio-9p,path=/Users/myoung/Library/Containers/com.docker.docker/Data/s40,tag=db -s 5,virtio-rnd -s 6,virtio-9p,path=/Users/myoung/Library/Containers/com.docker.docker/Data/s51,tag=port -s 7,virtio-sock,guest_cid=3,path=/Users/myoung/Library/Containers/com.docker.docker/Data,guest_forwards=2376;1525 -l com1,autopty=/Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty,log=/Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/console-ring -f kexec,/Applications/Docker.app/Contents/Resources/moby/vmlinuz64,/Applications/Docker.app/Contents/Resources/moby/initrd.img,earlyprintk=serial console=ttyS0 com.docker.driver="com.docker.driver.amd64-linux", com.docker.database="com.docker.driver.amd64-linux" ntp=gateway mobyplatform=mac -F /Users/myoung/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/hypervisor.pid
502 14559 14465   0 10:46PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.frontend {"action":"vmstateevent","args":{"vmstate":"running"}}
502 14560 14559   0 10:46PM ??         0:00.01 /Applications/Docker.app/Contents/MacOS/com.docker.frontend {"action":"vmstateevent","args":{"vmstate":"running"}}
502 14562 13146   0 10:46PM ttys000    0:00.00 grep -i com.docker

You should notice the Docker processes are running again. You can also check the timestamp of files in the Docker image directory on the external hard drive:

ls -la /Volumes/Samsung/Docker/image/com.docker.driver.amd64-linux
total 134133536
drwxr-xr-x@ 12 myoung  staff          408 Nov  7 22:42 .
drwxr-xr-x   3 myoung  staff          102 Nov  3 17:08 ..
-rw-r--r--   1 myoung  staff  68676222976 Nov  7 22:45 Docker.qcow2
-rw-r--r--   1 myoung  staff        65536 Nov  7 22:42 console-ring
-rw-r--r--   1 myoung  staff            5 Nov  7 22:42 hypervisor.pid
-rw-r--r--   1 myoung  staff            0 Aug 24 16:06 lock
drwxr-xr-x  67 myoung  staff         2278 Nov  5 22:00 log
-rw-r--r--   1 myoung  staff           17 Nov  7 22:42 mac.0
-rw-r--r--   1 myoung  staff           36 Aug 24 16:06 nic1.uuid
-rw-r--r--   1 myoung  staff            5 Nov  7 22:42 pid
-rw-r--r--   1 myoung  staff        59619 Nov  7 22:42 syslog
lrwxr-xr-x   1 myoung  staff           12 Nov  7 22:42 tty -> /dev/ttys001

You should notice the timestamp of the Docker.qcow2 file has been updated which means Docker is now using this location for its image file.

Start a Docker container

You should attempt to start a Docker container to make sure everything is working fine. You can start the HDP sandbox via docker start sandbox if you've already installed it as listed in the prerequisites. If everything is working fine, you can delete the backup.

Delete Docker backup image

Now that everything is working using the new location, we can remove our backup.

rm -rf ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux.backup

Review

If you successfully followed along with this tutorial, we were able to move our Docker for Mac virtual machine image to an external hard drive. This should free up to 64GB of space on your laptop hard drive. Look for part two in the series to learn how to increase the size of your Docker image.

7,501 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 08:26 AM
Updated by:
 
Contributors
Top Kudoed Authors