Support Questions
Find answers, ask questions, and share your expertise

How to increase the capacity of HDFS?

Expert Contributor

I am posting this answer after searching in the internet for a good explanation. Currently the total physical hard disk space (4 nodes) is 720 GB. The dashboard currently shows that only 119 GB is configured for DFS. I want to increase this space to at last 300 GB. I didn't find anything staright forward on Ambari dashboard to do this. The only information I found on the internet is to modifify core-site.xml file to hav a property hadoop.tmp.dir pr that points to another directory. I do not want to blankly do it, without understanding what it means to be expanding HDFS capacity and how to do it through Ambari Dashboard.

1 ACCEPTED SOLUTION

Mentor

You add capacity by giving dfs.datanode.data.dir more mount points or directories. In Ambari that section of configs is I believe to the right depending the version of Ambari or in advanced section, the property is in hdfs-site.xml. the more new disk you provide through comma separated list the more capacity you will have. Preferably every machine should have same disk and mount point structure

View solution in original post

17 REPLIES 17

Mentor

You add capacity by giving dfs.datanode.data.dir more mount points or directories. In Ambari that section of configs is I believe to the right depending the version of Ambari or in advanced section, the property is in hdfs-site.xml. the more new disk you provide through comma separated list the more capacity you will have. Preferably every machine should have same disk and mount point structure

Expert Contributor

@Artem Ervits Can you please elaborate what you what you mean by "right spending of version of ambari". I checked "Advanced hdfs-site" section, but I dont see any "dfs.datanode.data.dir"

Mentor

sorry auto-correct on my tablet. @Pradeep kumar I updated the answer with correct spelling.

Expert Contributor

@Artem Ervits. Thanks, but I still could not find this property under "Advnce hdfs-site" section. I was reading the link provided by Neeraj Sabharwal, in his answer below, which also talks about mentioning /hadoop as the folder in the property 'dfs.datanode.data.dir'. But, like I said, I could not find this property.

Expert Contributor

@Artem Ervits I found "Data Node Directories" under "Data Node" section under "Settings" tab. The "Data Node Directories" has the folder name /hadoop/hdfs/data. However, when I do df -h, I do not see this folder in the mount information. Following is the output of my the df -h on the master server:

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/vg_item70288-lv_root 50G 41G 6.2G 87% /

tmpfs 3.8G 0 3.8G 0% /dev/shm

/dev/sda1 477M 67M 385M 15% /boot

/dev/mapper/vg_item70288-lv_home 172G 21G 143G 13% /home

Mentor

That's the problem, you need to replace that path with correct path that does exist, otherwise data is being written to filesystem and run out of space quickly

Expert Contributor

@Artem Ervits: I am having many issues now. 1) Ambari doesn't allow me to remove folder name "/hadoop/hdfs/data". So I cannot completely replace it with a new folder. 2) If I give /hadoop/hdfs/data,/home then it shows me error Can't start with "home(s)". I am pretty sure something is wrong.

Mentor

Create mount point /hadoop/ pointing to your large disk

Expert Contributor

@Artem Ervits: Okay. I have finally got what I wanted and I have increased the DFS capacity. Thanks for your help. I learned a lot through this exercise :). I am accepting your answer and also providing steps that I followed in another answer post, so that it will be helpful to other users.

Mentor

Excellent, glad it worked

Expert Contributor

@Neeraj Sabharwal I deleted my previous comment as it didn't make any sense. What I currently don't understand is that, the "DataNode directories" show /hadoop/hdfs/data. I am not able to change this. If I edit the field to remove this folder name, the "save" button gets disabled. It is not taking /home folder as a valid folder. The /home mount has the maximum space, and I am not able to mention this in the "DataNode directories" field. Any ideas?. Thanks.

Expert Contributor

I am posting this so that it will be helpful for those users who are looking towards understanding how DFS capacity could be increased. I am providing the details in steps below.

1) The section "HDFS Disk usage" (a box) on the dashboard, shows the current DFS usage. However, the total DFS capacity is not shown here.

2) To view the total capacity use Name Node Web UI eg. (http://172.26.180.6:50070/). This will show you the total DFS capacity.

3) It is helpful to see the file system information by executing "df -h", which tells you the size of the file system. In my case the root file system had very less space allocated (50 GB) to it as compared to file system mounted on /home (750 GB).

4) The straight forward way to increase the DFS capacity is mention additional folder in the "DataNode directories" field under HDFS -> Configs -> Settings tab, as a comma separated value. This new folder should exist in a file system that has more disk capacity.

5) Ambari for some reason does not accept /home as the folder name for storing file blocks. By default it shows "/hadoop/hdfs/data. You cannot delete it completely to replace it with new folder path.

6) The best way is to create a new mount point and point it to a folder in the /home. Therefore create a mount point eg. Hdfsdata and then point it to a folder under home, eg. /home/hdfsdata. Following are the steps to create a new mount point:

  1. Create a folder in the root eg. /hdfsdata Create a folder under home eg. /home/hdfsdata
  2. Provide permission to 'hdfs' user to this folder: chown hdfs:hadoop -R /home/hdfsdata
  3. Provide file/folder permissions to this folder: chmod 777 -R /home/hdfsdata.
  4. Mount this new folder mount --bind /home/hdfsdata/ /hdfsdata/

After the above steps, restart the HDFS service and you have your capacity increased.

I have done all the steps you have given above and I am facing an issue right now while restarting the HDFS service. Here is the log attached below.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 167, in <module>
    DataNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 530, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 62, in start
    datanode(action="start")
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py", line 72, in datanode
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 267, in service
    Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-hadoop1ind1.india.out
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000bc800000, 864026624, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 864026624 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /var/log/hadoop/hdfs/hs_err_pid51884.log

Can you please look out at and tell me what exactly went wrong ?

Previous comment was for namenode restart and this is I am showing you the datanode restart after allocating more memory to java.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 167, in <module>
    DataNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 530, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 62, in start
    datanode(action="start")
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py", line 72, in datanode
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 267, in service
    Execute(daemon_cmd, not_if=process_id_exists_command, environment=hadoop_env_exports)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-hadoop1ind1.india.out

@Pradeep kumar : Can you please have a look at both the log and help me out extending my current HDFS storage

New Contributor

I had the same alert. The capacity of the DN(datanode) was somehow assigned to the very small space. After reading the threads here, I was about to create a partition and mount it on a new DN directory. However, since I did not have any issue to use the /hadoop/hdfs/data which is in /(root) directory, I tried to find other way around and found that the amount of "Reserved space for HDFS" under the advanced tab was huge almost take whole unused space of the root directory. After downsize of "Reserved space for HDFS", every alert was solved.

Expert Contributor

That is a good point David Hwang. Thanks for sharing 🙂

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.