Reply
Explorer
Posts: 11
Registered: ‎09-20-2017

clusterdock: how to increase storage space on data nodes

[ Edited ]

Hello,

 

I have a CentOS host with 64GB RAM and 1TB HDD. In there I created a cluster following this guide with 1 master and 1 secondary node.

 

Now, it seems that docker has allocated 50GB disk space for the data node (from which 32GB are reserved for swapping). How can I increase the 50GB?

 

Or, how can I mount dfs.data.dir to a host's volume?

 

Thank you,

Gerasimos

 

P.S. Couldn't find a better Label than 'HDFS'...

Posts: 390
Topics: 11
Kudos: 60
Solutions: 35
Registered: ‎09-02-2016

Re: clusterdock: how to increase storage space on data nodes

@gerasimos

 

if you have additional file system in your node then you can create a folder for datanode (ex: /hdfs1/dfs/dn) and go to CM -> HDFS -> Configuration -> search for "DataNode Data Directory" -> link the above path here. 

Explorer
Posts: 11
Registered: ‎09-20-2017

Re: clusterdock: how to increase storage space on data nodes

Thank you for the reply. The problem is that the additional space is on the host CentOS machine file system and not allocated or mounted on the docker container of the data node.

 

I need to mount this space to the docker container (which was initiated by clusterdock).

 

Thank you,

Gerasimos

New Contributor
Posts: 7
Registered: ‎10-31-2017

Re: clusterdock: how to increase storage space on data nodes

The issue your having is with Docker's use of the devicemapper storage driver. Doing a web search should turn up some resources on how to increase the amount of space allocated for each container, though I'd personally suggest running Ubuntu or another operating system that supports overlayfs, which is significantly less burdensome to use with Docker.

Announcements