question Re: Docker hdp 2.5 import is very slow in Archives of Support Questions (Read Only)

Docker hdp 2.5 import is very slow

malderhout — Mon, 26 Sep 2016 14:39:21 GMT

Great that docker is supported with HDP

When I import the docker hdp 2.5 tar file the command, will never stop

$ docker load < HDP_2.5_docker.tar.gz

I hear my Macbook is running, but this proces is very slow

Maybe it takes some time 🙂 (the import in Virtualbox is much faster)

I use Docker version 1.12

Greets,

Maikel

Re: Docker hdp 2.5 import is very slow

myoung — Sat, 01 Oct 2016 20:12:47 GMT

@Maikel Alderhout

The docker load command does take some time as the docker file is ~14GB in size. However, it looks like you are trying to load the tar.gz file from your example above. Before running the docker load command, you must first gunzip the HDP_2.5_docker.tar.gz file.

$ gunzip HDP_2.5_docker.tar.gz
$ docker load < HDP_2.5_docker.tar

Re: Docker hdp 2.5 import is very slow

avijeetd — Tue, 25 Oct 2016 20:23:10 GMT

@Michael Young

Hi Michel,

while doing docker load, it ran out of space

Is it possible to tell it load and use a specific location? Also how can I clean the half loaded image.

Thanks

Avijeet

b1b065555b8a: Loading layer 202.2 MB/202.2 MB

0b547722f59f: Loading layer 10.36 GB/13.84 GB

ApplyLayer exit status 1 stdout: stderr: write /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/lib/rt.jar: no space left on device

Re: Docker hdp 2.5 import is very slow

myoung — Wed, 26 Oct 2016 01:56:17 GMT

@Avijeet Dash

When you run the docker load command, it's copying the container image to the virtual machine that Docker uses to run the containers. You can't change the location of where those images are stored that I know of.

You can run docker images to see a list of images that have been loaded. You may see something like this:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
sandbox             latest              09252f3bc286        2 weeks ago         13.84 GB
elasticsearch       latest              22287ab1f811        3 weeks ago         342.8 MB
kibana              latest              67c20a93c5bc        4 weeks ago         297 MB
alpine              edge                e4c65b272e02        4 weeks ago         4.824 MB
<none>              <none>              fc813bdc4bdd        4 weeks ago         14.57 GB
hdp/postgres        latest              26e6495659e4        5 weeks ago         310.9 MB
centos              6                   f07f6ca555a5        6 weeks ago         194.6 MB
postgres            latest              6f86882e145d        7 weeks ago         265.9 MB

You can delete any images that you don't need with docker rmi <image id>. You can also use docker ps -a to see a list of containers that are lying around. You may see something like this:

 docker ps -a
CONTAINER ID        IMAGE               COMMAND               CREATED             STATUS                     PORTS               NAMES
15411dc968ad        fc813bdc4bdd        "/usr/sbin/sshd -D"   2 weeks ago         Exited (255) 3 hours ago                       hdp25-atlas-demo

If you have a lot of containers sitting around, they do take up space. You can remove any containers you don't need by using docker rm <container id>.

Re: Docker hdp 2.5 import is very slow

avijeetd — Wed, 26 Oct 2016 12:12:14 GMT

Hi @Michael Young

I don't have any other images, this is the only image i was trying to load

I have a azure centos vm, which root directory / has only 30GB

I have another directory with has 500Gb

but wondering how to tell docker to use that directory

thanks,

Avijeet

Re: Docker hdp 2.5 import is very slow

maruykr1981 — Wed, 09 Nov 2016 18:38:58 GMT

Hi @Avijeet Dash

Did you solve the problem ? Now, I'm facing the same problem.

I have a aws vm. (m4.xlarge: centos7)

I don't have any docker image.

$ sudo docker info
Containers: 0
Running: 0 
Paused: 0 
Stopped: 0
Images: 0
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker-202:80-17039361-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.8 MB
 Data Space Total: 107.4 GB
 Data Space Available: 107.4 GB
 Metadata Space Used: 581.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /docker-image/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /docker-image/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.389 GiB
Name: ip-xxx-xx-xx-xx.ap-northeast-1.compute.internal
ID: xxx:xxx:xxx
Docker Root Dir: /docker-image
Debug Mode (client): false
Debug Mode (server): false
Registry: <a href="https://index.docker.io/v1/">https://index.docker.io/v1/
</a>WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

I changed the image location from "/var/lib/docker" to "/docker-image".

"/docker-image" is a separate disk from "/var/lib/docker".

However, I can't docker load, it ran out of space.

Please, tell me the next actions to solve it.

thanks.

Re: Docker hdp 2.5 import is very slow

myoung — Thu, 10 Nov 2016 01:44:38 GMT

@Atsushi Marumo

I have written an article guiding you through the process with CentOS 7. This is the preferred way to do this: https://community.hortonworks.com/content/kbentry/65714/how-to-modify-the-default-docker-configuration-on.html

You can read more about CentOS 7 and Docker image location here: https://docs.docker.com/engine/admin/systemd/

Re: Docker hdp 2.5 import is very slow

maruykr1981 — Fri, 11 Nov 2016 00:45:29 GMT

Hi @Michael Young

Thank you, I have solved the problem by your answer.

The cause of the problem, seems to be that "dm.basesize" is smaller.

I didn't change the image location and the storage-driver.

I was only change the "dm.basesize".

Thanks.