Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

clusterdock: how to increase storage space on data nodes

Highlighted

clusterdock: how to increase storage space on data nodes

Rising Star

Hello,

 

I have a CentOS host with 64GB RAM and 1TB HDD. In there I created a cluster following this guide with 1 master and 1 secondary node.

 

Now, it seems that docker has allocated 50GB disk space for the data node (from which 32GB are reserved for swapping). How can I increase the 50GB?

 

Or, how can I mount dfs.data.dir to a host's volume?

 

Thank you,

Gerasimos

 

P.S. Couldn't find a better Label than 'HDFS'...

3 REPLIES 3
Highlighted

Re: clusterdock: how to increase storage space on data nodes

Champion

@gerasimos

 

if you have additional file system in your node then you can create a folder for datanode (ex: /hdfs1/dfs/dn) and go to CM -> HDFS -> Configuration -> search for "DataNode Data Directory" -> link the above path here. 

Highlighted

Re: clusterdock: how to increase storage space on data nodes

Rising Star

Thank you for the reply. The problem is that the additional space is on the host CentOS machine file system and not allocated or mounted on the docker container of the data node.

 

I need to mount this space to the docker container (which was initiated by clusterdock).

 

Thank you,

Gerasimos

Re: clusterdock: how to increase storage space on data nodes

Explorer

The issue your having is with Docker's use of the devicemapper storage driver. Doing a web search should turn up some resources on how to increase the amount of space allocated for each container, though I'd personally suggest running Ubuntu or another operating system that supports overlayfs, which is significantly less burdensome to use with Docker.

Don't have an account?
Coming from Hortonworks? Activate your account here