question Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster in Support Questions

Increasing the HDFS Disk size in HDP 2.3 3 node cluster

Kunal.Gaikwad — Sun, 18 Aug 2019 12:06:38 GMT

In the 3 node cluster installation for POC, My 3rd note is datanode, it has a disk space of about 200 GB.

As per the widget, my current HDFS Usage is as follows:

DFS Used: 512.8 MB (1.02%); non DFS used 8.1 GB (16.52%); remaining 40.4GB (82.46 %)

When I do df -h to check the disk size i can see a lot of space is taken by tmpfs as shown in the following screenshot:

How can I increase my HDFS disk size?

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

nsabharwal — Thu, 25 Feb 2016 19:49:55 GMT

@Kunal Gaikwad

tmpfs is out of pic.

You need to increase /home space ...Is it a vm? If yes then you can reach out to system admin to allocate more space to /home

If it's bare metal then attach another disk and make it part of your cluster.

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

nsabharwal — Thu, 25 Feb 2016 19:53:34 GMT

vm https://www.rootusers.com/how-to-increase-the-size-of-a-linux-lvm-by-expanding-the-virtual-machine-disk/

lvm --> http://www.linuxuser.co.uk/features/resize-your-disks-on-the-fly-with-lvm

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

Kunal.Gaikwad — Thu, 25 Feb 2016 19:55:26 GMT

Yes the cluster is on VM Vsphere

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

nsabharwal — Thu, 25 Feb 2016 20:06:14 GMT

@Kunal Gaikwad That's easy then. See the link that I shared above.

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

Kunal.Gaikwad — Thu, 25 Feb 2016 20:22:03 GMT

My /home partition is already 130 GB, but I am not able to use it for HDFS as metioned above as per the gadget. My concern is it should not hamper my HDP installation which I have already done on it

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

nsabharwal — Thu, 25 Feb 2016 20:42:35 GMT

@Kunal Gaikwad I believe you are using / for hadoop..Is that correct?

Generally, extending / is complicated than non-root

Install will be fine as long as there is no human error

If possible then you can add 4th nodes in the cluster and decomsission the 3rd one

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

Kunal.Gaikwad — Thu, 25 Feb 2016 20:48:25 GMT

Yes i am using / for hadoop. This partitions are done automatically by ambari, so want to increase the HDFS size, for the POC wanted to import a table of size 70 GB, but because of the current HDFS size, I am able to import only 30+ GB's and the job gets hanged with alerts all over the ambari about the disk usage.

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

nsabharwal — Thu, 25 Feb 2016 20:55:42 GMT

@Kunal Gaikwad that's why you have to be careful about giving mounts during the install. If possible then add new nodes and make sure to add new disk mounts in the configs.

or extend the existing LUNS

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

Kunal.Gaikwad — Fri, 26 Feb 2016 17:45:53 GMT

I am trying to upgrade the storage i have taken the snapshot of the cluster. Just to be sure I need to increase /dev/mappper/centos-root storage size right?

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

nsabharwal — Fri, 26 Feb 2016 18:06:35 GMT

@Kunal Gaikwad

Yes as its your root where you have hadoop data

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

nsabharwal — Sun, 28 Feb 2016 21:07:30 GMT

@Kunal Gaikwad Did it work out fine?

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

vinaykumarpotnu — Mon, 07 Mar 2016 14:52:21 GMT

How did you solve this problem ?

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

Kunal.Gaikwad — Wed, 16 Mar 2016 20:54:01 GMT

yes we did increase the root space without hampering the installation, we did follow simple process of increasing the size. we created a external virtual space and merged with root

Re: Increasing the HDFS Disk size in HDP 2.3 3 node cluster

chahrimourad — Wed, 21 Sep 2016 21:29:05 GMT

how to increase it for no vm please @Neeraj Sabharwal