question Re: Configure Storage capacity of Hadoop cluster in Archives of Support Questions (Read Only)

Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Mon, 19 Aug 2019 11:17:42 GMT

we have 5 node cluster with following configurations for master and slaves.

HDPMaster   35 GB   500 GB
HDPSlave1   15 GB   500 GB
HDPSlave2   15 GB   500 GB
HDPSlave3   15 GB   500 GB
HDPSlave4   15 GB   500 GB
HDPSlave5   15 GB   500 GB

But the cluster is not taking much space. I am aware of the fact that it will reserve some space for non-dfs use.But,it is taking uneven capacity for each slave node. Is there a way to reconfigure hdfs ?

PFA.

Even though all the nodes have same hard disk, only slave 4 is taking 431GB, remaining all nodes are utilizing very small space. Is there a way to resolve this ?

Re: Configure Storage capacity of Hadoop cluster

aervits — Fri, 04 Mar 2016 18:28:30 GMT

Is your replication factor set to 3? Are you using one reducer in your ingestion? You can use hdfs balancer to spread the data around your cluster https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#Administration_Commands

Re: Configure Storage capacity of Hadoop cluster

aervits — Fri, 04 Mar 2016 18:31:17 GMT

I think you're interpreting it wrong it's the opposite, only slave 4 is not taking up data, the other nodes are filled.

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Fri, 04 Mar 2016 18:32:04 GMT

yes, Replication factor is 3. But how spreading the data around the cluster help us in changing capacity of the nodes ?

Re: Configure Storage capacity of Hadoop cluster

aervits — Fri, 04 Mar 2016 18:36:05 GMT

That's remaining capacity not total

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Fri, 04 Mar 2016 18:39:14 GMT

Yeah probably. I am new to this and I am not able to understand this whole configuration thing. Capacity is available space and non dfs is the space available for linux system use, if i am not wrong. so I still didn't understand the answer to my question. Why is the capacity(the available space ) is more for slave 4 alone when all the nodes including master have the same harddisk capacity.

Re: Configure Storage capacity of Hadoop cluster

aervits — Fri, 04 Mar 2016 18:56:52 GMT

Go to the node and investigate the data dir directory you specified. Run hdfs fsck / command see if you have issue with hdfs, post screenshot of main page of Ambari with all widgets.

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Fri, 04 Mar 2016 20:23:38 GMT

Cluster is new. It hardly contain any data in it.

Re: Configure Storage capacity of Hadoop cluster

aervits — Fri, 04 Mar 2016 21:10:54 GMT

OK you need to confirm which directories you specified for datanode in Ambari > hdfs > configs

Re: Configure Storage capacity of Hadoop cluster

nsabharwal — Sat, 05 Mar 2016 18:26:43 GMT

@vinay kumar

I have never seen the same number for all the slave nodes because of the data distribution.

Link

To overcome uneven block distribution scenario across the cluster, a utility program called balancer

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#balancer

Re: Configure Storage capacity of Hadoop cluster

ahadjidj — Sat, 05 Mar 2016 23:00:38 GMT

@vinay kumar

Maybe you have problem in disk partitioning. Can you check how much space you have allocated for partitions used by HDP?

Here's a link for partitioning recommendations http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_cluster-planning-guide/content/ch_partitioning_chapter.html

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Mon, 07 Mar 2016 13:55:08 GMT

/opt/hadoop/hdfs/data,/tmp/hadoop/hdfs/data,/usr/hadoop/hdfs/data,/usr/local/hadoop/hdfs/data

Re: Configure Storage capacity of Hadoop cluster

aervits — Mon, 07 Mar 2016 14:07:24 GMT

You need to pick just one there, especially don't choose /tmp as your parent dir, asking for trouble

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Mon, 19 Aug 2019 11:17:34 GMT

I will make it clear.The Cluster is new and it don't have much data in it. As per my understanding, Available capacity is the storage available for data node(hdfs) , if i am not wrong. The actual hard disk size of each node being 500 GB and the available capacity for 5 of them is far to less than the slave 4.root folder disk capacity has more than 400gb space allocated and the same should be allocated to hdfs. My concern is where did the rest of the space go? How did the distribution thing come here when my only concern is about hdfs capacity. PFA.

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Mon, 07 Mar 2016 14:27:26 GMT

Like you said, i removed /tmp from that directories list and the capacity of all the nodes reduced to 40 gb or less including slave 4

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Mon, 19 Aug 2019 11:17:21 GMT

I have allocated around 400 GB for / partition. PFA.

Re: Configure Storage capacity of Hadoop cluster

nsabharwal — Mon, 07 Mar 2016 17:23:33 GMT

@vinay kumar What do you have for dfs.datanode.data.dir?

slave 4 --> / ?

Rest of the nodes are using other mounts...

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Mon, 07 Mar 2016 17:43:09 GMT

dfs.datanode.data.dir have: /opt/hadoop/hdfs/data,/tmp/hadoop/hdfs/data,/usr/hadoop/hdfs/data,/usr/local/hadoop/hdfs/data

All the nodes have same mounts.

Re: Configure Storage capacity of Hadoop cluster

vinaykumarpotnu — Mon, 07 Mar 2016 17:46:49 GMT

@Neeraj Sabharwal Did say anything wrong ? So the capacity is the Space allocated to hdfs right?

Re: Configure Storage capacity of Hadoop cluster

nsabharwal — Mon, 07 Mar 2016 17:57:10 GMT

@vinay kumar

As expected, the problem is with the disks allocated to datanode settings.

Ambari picks up all the mounts except /boot and /mnt

You were suppose to modify the settings during the installs. As you can see, data is going on /opt and other mounts and you were suppose to give only /hadoop " / has 400GB"

Now , there is no way we want to store the data on /tmp

/opt/hadoop/hdfs/data,/tmp/hadoop/hdfs/data,/usr/hadoop/hdfs/data,/usr/local/hadoop/hdfs/data

You need to create a directory as /hadoop and modify the settings to read the data from /hadoop.