Support Questions
Find answers, ask questions, and share your expertise

Namenode data directory size problem

Namenode data directory size problem

Explorer

Hello,

 

I installed a CDH5 cluster using 2 EC2 instances on Ubuntu 12.04 LTS 64 bit. I used the cloudera manager installer to point to two instances I setup with 50GB of space each (prototyping here). I set the ssh keys up and everything was good going through the wizard. I put the name node on the node without cloudera manager, and the secondary name node on cloudera manager.

 

After installation, I get this critical error from the active name node (the one without cloudera manager installed):

 

The following data directories are on filesystems with less than 5.0 GiB of their space free. /mnt/dfs/nn (free: 3.7 GiB (93.07%), capacity: 3.9 GiB) All data directories (/dfs/nn) are on filesystems with more than 10.0 GiB of their space free.

 

The cluster says it has 88 GB free, what do I need to do to allocate more space to this mount point?

 

Thank you,

 

Jayme

6 REPLIES 6

Re: Namenode data directory size problem

Please post the output of "df -h" and "/usr/sbin/mount" from both hosts

Regards,
Gautam Gopalakrishnan

Re: Namenode data directory size problem

Explorer

df -h on host 1:

 

Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 50G 8.4G 39G 18% /
udev 1.9G 12K 1.9G 1% /dev
tmpfs 377M 276K 377M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.9G 0 1.9G 0% /run/shm
/dev/xvdb 4.0G 73M 3.7G 2% /mnt
cm_processes 1.9G 2.2M 1.9G 1% /run/cloudera-scm-agent/process

df -h on host 2:

 

/dev/xvda1 50G 6.7G 41G 15% /
udev 1.9G 8.0K 1.9G 1% /dev
tmpfs 377M 276K 377M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.9G 0 1.9G 0% /run/shm
/dev/xvdb 4.0G 75M 3.7G 2% /mnt
cm_processes 1.9G 1.8M 1.9G 1% /run/cloudera-scm-agent/process

 

mount on host 1:

 

/dev/xvda1 on / type ext4 (rw,discard)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/xvdb on /mnt type ext3 (rw)
cm_processes on /run/cloudera-scm-agent/process type tmpfs (rw,mode=0751)

 

mount on host 2:

 

proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/xvdb on /mnt type ext3 (rw)
cm_processes on /run/cloudera-scm-agent/process type tmpfs (rw,mode=0751)

Re: Namenode data directory size problem

Cloudera Manager is complaining about /mnt/dfs/nn having less tha n5.0GB whereas /dfs/nn has more than than 5.0GB free.


/dev/xvdb 4.0G 73M 3.7G 2% /mnt

 

Please check your HDFS configuration for the edits directory. Just visit the HDFS service, open the configuration and search for /mnt. You should see where it is being used.

Regards,
Gautam Gopalakrishnan

Re: Namenode data directory size problem

Explorer

Looking at the HDFS configuration, there are two directories listed for NameNode data:

 

/dfs/nn

/mnt/dfs/nn

 

This was setup by default by the installer. What do I need to do to fix the critical error at this point?

 

Thanks for your help so far.

 

Jayme

Re: Namenode data directory size problem

Explorer
P.S. I also see the /mnt directory being used in "DataNode Default Group" (/mnt/dfs/dn) and "DataNode Group 1" (/mnt/dfs/dn) again.

Re: Namenode data directory size problem

The installer generally adds two directories for the namenode edits
directory to ensure two copies exist. In your case /mnt has very little
space and this is triggering the warning. If this is a test cluster you're
fine with just one edits directory. So you can delete /mnt/dfs/nn and
/mnt/dfs/dn from the configuration and restart HDFS. Do keep in mind that
if the sole copy gets corrupted somehow, you'll lose all your contents.

Regards,
Gautam Gopalakrishnan