Increase size of HDFS.

Question

I am running Hortonworks Sandbox HDP 2.6.5 on VirtualBox. I have increased the size of my virtual hard disk (.vdi) to 500 GB. However, when I login to Ambari and view the size of my disk, it shows 106 GB only. What should I do to increase the HDFS capacity from 106 GB to 500 GB?

focal_fossa · Accepted Answer

Since the solution is scattered across many posts, I'm posting a short summary of what I did.

I am running HDP 2.6.5 image on VirtualBox.

Increased my virtual hard disk through Virtual Media Manager
In the guest OS,
1. Partitioned the unused space
2. Formatted the new partition as an ext4 file system
3. Mounted the file system
4. Update the /etc/fstab (I couldn't do it, as I did not find that file
In Ambari, under DataNode directory config, added the newly mounted file system as a comma separated value
Restarted HDFS (my cluster did not have any files, therefore I did not run the below)

Thanks to @Shelton for his guidance.

sudo -u hdfs hdfs balancer

Shelton · Answer

@focal_fossa

AFAIK these sandboxes dynamically allocated storage. You can try that by generate and load data for TPC-DS

General usage is

tpcds-setup.sh scale_factor [directory]

For example below will generate 200 GB of TPC-DS data in /user/data [HDFS]

./tpcds-setup.sh 200  /user/data

This should prove that the disk allocation is dynamic see below links

https://github.com/hortonworks/hive-testbench/blob/hive14/tpch-build.sh and https://github.com/hortonworks/hive-testbench/blob/hive14/tpch-setup.sh to build

Hope that helps

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.