Reply
New Contributor
Posts: 3
Registered: ‎08-11-2013

cloudera VM enlarge the HDFS FS capacity

Hi,

 

I've installed the VM (on VirtualBox) and successfully started working.

The FS is limited to 11 GB as shown:

[cloudera@localhost ~]$ hadoop fs -df -h
Filesystem Size Used Available Use%
hdfs://localhost.localdomain:8020 11.1 G 59.0 M 7.9 G 1%

 

how can I enlarge the HDFS FS capacity to 50GB?

 

thanks,

Nadav

Highlighted
Expert Contributor
Posts: 63
Registered: ‎08-06-2013

Re: cloudera VM enlarge the HDFS FS capacity

Add more volumes/disks to the dfs.datanode.data.dir in hdfs-site.xml to increase the HDFS capacity.

Posts: 1,568
Kudos: 293
Solutions: 240
Registered: ‎07-31-2013

Re: cloudera VM enlarge the HDFS FS capacity

HDFS's space comes out of what local HDD space is available to it. Within the QuickStart VMs, HDFS would use the same disk as the VM itself, so to expand the HDFS DataNode's storage, you'd simply have to raise the VM's available space configuration.

 

For VirtualBox users, instructions on raising space can be found at http://jonmifsud.com/blog/increase-virtualbox-disk-size/ (You can google for other such articles on VirtualBox to troubleshoot further).

Backline Customer Operations Engineer
sph
Explorer
Posts: 10
Registered: ‎03-15-2016

Re: cloudera VM enlarge the HDFS FS capacity

[ Edited ]

If you are using a docker image and Quickstart VM is via Virtual Box. You can use following command to increase your disk space.

1. Stop the running VM (say default)

2. Delete default vm.

3. Issue a command docker-machine create -d virtualbox --virtualbox-disk-size "50000" default   ......(50GB set here)

4. Start the default VM

5. Start the docker terminal

 

If #3 gives you any error saying default (or your machine) already exist, then go to the folder and delete the docker machine.

Explorer
Posts: 12
Registered: ‎05-25-2016

Re: cloudera VM enlarge the HDFS FS capacity

For VMware with CentOS this should work:

 

A) Edit VMWare settings and resources:

 

Edit Settings in VMWare workstation and assign 

 

1) Memory: 10+ GB

2) Processors : 2

3) Hard Disk : 100GB or as per your choice

 

B) Create a new Partition and add it to the Volume Group in Single user mode

 

Then boot the operating system in the Single User mode by following steps : 

 

1) Start the machine and wait for the boot screen, then press any key

2) You will see the CentOS operating system, press 'e'

3) Then you will see 3 entries of root, kernel and initrd, Select kernel and press 'e' again

4) Modify the entry followed by "single single" in the last line of the file and press Enter

5) then you'll be back to the 3 entries and press 'b' for boot

6) You'll be now in the Single User mode

 

 

Let's create one more /dev/sda3, Then Do

 

fdisk /dev/sda

Then in the fdisk do

 

  p (to see existing partitions)
  n (new partition)         
      p (primary partition)         
      When asked "Partition number (1-4)", enter "3".       
      accept defaults.  It will use the rest of the disk    
  w (write changes)

 

Then Reboot the machine & follow the steps above for the Single user mode again: 

 

init 6 or reboot

Create a physical volume and extend the VolumeGroup using :

 

pvcreate /dev/sda3
vgdisplay //Get the name of the volume group. vg_quickstart will be in the cloudera's VM
vgextend vg_quickstart /dev/sda3
lvextend /dev/vg_quickstart/lv_root -r -l+100%FREE //(note : -r option will format it too.)

 

Then reboot without single user mode and wait for 15 mins as it'll take time to boot.

 

Done!

Announcements