11-02-2017 03:32 AM - edited 11-02-2017 04:22 AM
I have a CentOS host with 64GB RAM and 1TB HDD. In there I created a cluster following this guide with 1 master and 1 secondary node.
Now, it seems that docker has allocated 50GB disk space for the data node (from which 32GB are reserved for swapping). How can I increase the 50GB?
Or, how can I mount dfs.data.dir to a host's volume?
P.S. Couldn't find a better Label than 'HDFS'...
11-02-2017 09:05 AM
if you have additional file system in your node then you can create a folder for datanode (ex: /hdfs1/dfs/dn) and go to CM -> HDFS -> Configuration -> search for "DataNode Data Directory" -> link the above path here.
11-02-2017 09:16 AM
Thank you for the reply. The problem is that the additional space is on the host CentOS machine file system and not allocated or mounted on the docker container of the data node.
I need to mount this space to the docker container (which was initiated by clusterdock).
11-03-2017 09:13 AM
The issue your having is with Docker's use of the devicemapper storage driver. Doing a web search should turn up some resources on how to increase the amount of space allocated for each container, though I'd personally suggest running Ubuntu or another operating system that supports overlayfs, which is significantly less burdensome to use with Docker.