Reply
New Contributor
Posts: 4
Registered: ‎05-25-2017

during cdsw init --> ERROR:: Unable to start docker: 1

[ Edited ]

Hi,

 

I am not sure why cdsw is not finishing the initialization on my system.

 

this is the output of the init command:

[root@cdh-node4 ~]# cdsw init
Using user-specified config file: /etc/cdsw/config/cdsw.conf
Prechecking OS Version........[OK]
Prechecking scaling limits for processes........[OK]
Prechecking scaling limits for open files........[OK]
Prechecking that iptables are not configured........[OK]
Prechecking that SELinux is disabled........[OK]
Prechecking configured block devices and mountpoints........[OK]
Prechecking kernel parameters........[OK]
Prechecking that docker block devices are of adequate size........[OK]
Prechecking that application block devices are of adequate size........[OK]
Prechecking size of root volume........[OK]
Prechecking that CDH gateway roles are configured........[OK]
Prechecking that /etc/krb5 file is not a placeholder........
WARNING: The Kerberos configuration file [/etc/krb5.conf] seems to be a placeholder. If your CDH cluster is Kerberized, please copy /etc/krb5.conf to these Cloudera Data Science Workbench nodes.
Press enter to continue.
Prechecking parcel paths........[OK]
Prechecking CDH client configurations........[OK]
Prechecking Java version........[OK]
Prechecking Java distribution........[OK]
Creating docker thinpool if it does not exist
--- Logical volume ---
LV Name thinpool
VG Name docker
LV UUID lx938X-9syK-1m0E-61QW-XRzv-nRM5-euAEBb
LV Write Access read/write
LV Creation host, time cdsw-node1.novalocal, 2017-06-14 15:52:11 +1000
LV Pool metadata thinpool_tmeta
LV Pool data thinpool_tdata
LV Status available
# open 0
LV Size 665.00 GiB
Allocated pool data 1.20%
Allocated metadata 0.06%
Current LE 170239
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 252:2
Docker thinpool already configured.
Initialize application storage at /var/lib/cdsw
Disabling node with IP [192.168.1.110]...
Node [192.168.1.110] removed from nfs export list successfully.
Stopping rpc-statd...
Stopping nfs-idmapd...
Stopping rpcbind...
Stopping nfs-server...
Removing entry from /etc/fstab...
Unmounting [/dev/vdc1]...
Skipping format since volumes are already set correctly.
Adding entry to /etc/fstab...
Mounting [/var/lib/cdsw]...
Starting rpc-statd...
Enabling rpc-statd...
Starting nfs-idmapd...
Enabling nfs-idmapd...
Starting rpcbind...
Enabling rpcbind...
Starting nfs-server...
Enabling nfs-server...
Enabling node with IP [192.168.1.110]...
Node [192.168.1.110] added to nfs export list successfully.
Starting rpc-statd...
Enabling rpc-statd...
Starting nfs-idmapd...
Enabling nfs-idmapd...
Starting rpcbind...
Enabling rpcbind...
Starting nfs-server...
Enabling nfs-server...
Starting docker...
ERROR:: Unable to start docker: 1

 

OS release:

[root@cdh-node4 tmp]# cat /etc/*release
CentOS Linux release 7.2.1511 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.2.1511 (Core)
CentOS Linux release 7.2.1511 (Core)

 

This is whay I can see from journalctl when I try to start the docker service manually

 

Jun 16 12:53:53 cdh-node4.garvan.openstack.org systemd[1]: Starting docker...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has begun starting up.
Jun 16 12:53:53 cdh-node4.garvan.openstack.org docker[8302]: Command "daemon" is deprecated, and will be removed in Docker 1.16. Please run `dockerd` directly.
Jun 16 12:53:53 cdh-node4.garvan.openstack.org docker[8302]: time="2017-06-16T12:53:53.760406113+10:00" level=info msg="libcontainerd: new containerd process, pid: 8312"
Jun 16 12:53:54 cdh-node4.garvan.openstack.org docker[8302]: Error starting daemon: error initializing graphdriver: devmapper: Unable to take ownership of thin-pool (docker-thinpool) that already has used data blocks
Jun 16 12:53:54 cdh-node4.garvan.openstack.org systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jun 16 12:53:54 cdh-node4.garvan.openstack.org ifconfig[8330]: docker0: ERROR while getting interface flags: No such device
Jun 16 12:53:54 cdh-node4.garvan.openstack.org systemd[1]: docker.service: control process exited, code=exited status=255
Jun 16 12:53:54 cdh-node4.garvan.openstack.org systemd[1]: Failed to start docker.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Jun 16 12:53:54 cdh-node4.garvan.openstack.org systemd[1]: Dependency failed for Watches Docker for bad states..
-- Subject: Unit docker-monitor.timer has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-monitor.timer has failed.
--
-- The result is dependency.
Jun 16 12:53:54 cdh-node4.garvan.openstack.org systemd[1]: Job docker-monitor.timer/start failed with result 'dependency'.
Jun 16 12:53:54 cdh-node4.garvan.openstack.org systemd[1]: Unit docker.service entered failed state.
Jun 16 12:53:54 cdh-node4.garvan.openstack.org systemd[1]: docker.service failed.
Jun 16 12:53:57 cdh-node4.garvan.openstack.org polkitd[3385]: Registered Authentication Agent for unix-process:8337:69848 (system bus name :1.57 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent,
Jun 16 12:54:04 cdh-node4.garvan.openstack.org systemd[1]: docker.service holdoff time over, scheduling restart.
Jun 16 12:54:04 cdh-node4.garvan.openstack.org systemd[1]: Starting docker...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has begun starting up.
Jun 16 12:54:04 cdh-node4.garvan.openstack.org docker[8360]: Command "daemon" is deprecated, and will be removed in Docker 1.16. Please run `dockerd` directly.
Jun 16 12:54:05 cdh-node4.garvan.openstack.org docker[8360]: time="2017-06-16T12:54:05.014938273+10:00" level=info msg="libcontainerd: new containerd process, pid: 8376"
Jun 16 12:54:06 cdh-node4.garvan.openstack.org docker[8360]: Error starting daemon: error initializing graphdriver: devmapper: Unable to take ownership of thin-pool (docker-thinpool) that already has used data blocks
Jun 16 12:54:06 cdh-node4.garvan.openstack.org systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jun 16 12:54:06 cdh-node4.garvan.openstack.org ifconfig[8386]: docker0: ERROR while getting interface flags: No such device
Jun 16 12:54:06 cdh-node4.garvan.openstack.org systemd[1]: docker.service: control process exited, code=exited status=255
Jun 16 12:54:06 cdh-node4.garvan.openstack.org systemd[1]: Failed to start docker.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Jun 16 12:54:06 cdh-node4.garvan.openstack.org systemd[1]: Dependency failed for Watches Docker for bad states..
-- Subject: Unit docker-monitor.timer has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-monitor.timer has failed.
--
-- The result is dependency.
Jun 16 12:54:06 cdh-node4.garvan.openstack.org systemd[1]: Job docker-monitor.timer/start failed with result 'dependency'.
Jun 16 12:54:06 cdh-node4.garvan.openstack.org systemd[1]: Unit docker.service entered failed state.
Jun 16 12:54:06 cdh-node4.garvan.openstack.org systemd[1]: docker.service failed.
Jun 16 12:54:06 cdh-node4.garvan.openstack.org polkitd[3385]: Unregistered Authentication Agent for unix-process:8337:69848 (system bus name :1.57, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
~

 

Thank you

 

New Contributor
Posts: 4
Registered: ‎06-12-2017

Re: during cdsw init --> ERROR:: Unable to start docker: 1

 hi masber, the journalctl error logs say:

 

docker[8302]: Error starting daemon: error initializing graphdriver: devmapper: Unable to take ownership of thin-pool (docker-thinpool) that already has used data blocks

 

This is usually due to the fact that you are specifying a DOCKER_BLOCK_DEVICE that is not formatted and has some valid data blocks. Would it be possible to clear the devices that you have and rerun the operation?

New Contributor
Posts: 4
Registered: ‎06-12-2017

Re: during cdsw init --> ERROR:: Unable to start docker: 1

Also @masber, for CDSW v1.0.x, the quick way to cleanup docker is to run: /etc/cdsw/scripts/format-docker.sh. Please use a lot of care in using this command as it will unmount and delete content in the devices mentioned by DOCKER_BLOCK_DEVICES of /etc/cdsw/config/cdsw.conf.

 

Note that we can always get back the docker images, so if the above disks have been used only for docker, there is no issue.

 

Thanks