Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Namenode data directory size problem

Namenode data directory size problem

New Contributor

Hello,

 

I installed a CDH5 cluster using 2 EC2 instances on Ubuntu 12.04 LTS 64 bit. I used the cloudera manager installer to point to two instances I setup with 50GB of space each (prototyping here). I set the ssh keys up and everything was good going through the wizard. I put the name node on the node without cloudera manager, and the secondary name node on cloudera manager.

 

After installation, I get this critical error from the active name node (the one without cloudera manager installed):

 

The following data directories are on filesystems with less than 5.0 GiB of their space free. /mnt/dfs/nn (free: 3.7 GiB (93.07%), capacity: 3.9 GiB) All data directories (/dfs/nn) are on filesystems with more than 10.0 GiB of their space free.

 

The cluster says it has 88 GB free, what do I need to do to allocate more space to this mount point?

 

Thank you,

 

Jayme

6 REPLIES 6

Re: Namenode data directory size problem

Please post the output of "df -h" and "/usr/sbin/mount" from both hosts

Regards,
Gautam Gopalakrishnan

Re: Namenode data directory size problem

New Contributor

df -h on host 1:

 

Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 50G 8.4G 39G 18% /
udev 1.9G 12K 1.9G 1% /dev
tmpfs 377M 276K 377M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.9G 0 1.9G 0% /run/shm
/dev/xvdb 4.0G 73M 3.7G 2% /mnt
cm_processes 1.9G 2.2M 1.9G 1% /run/cloudera-scm-agent/process

df -h on host 2:

 

/dev/xvda1 50G 6.7G 41G 15% /
udev 1.9G 8.0K 1.9G 1% /dev
tmpfs 377M 276K 377M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.9G 0 1.9G 0% /run/shm
/dev/xvdb 4.0G 75M 3.7G 2% /mnt
cm_processes 1.9G 1.8M 1.9G 1% /run/cloudera-scm-agent/process

 

mount on host 1:

 

/dev/xvda1 on / type ext4 (rw,discard)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/xvdb on /mnt type ext3 (rw)
cm_processes on /run/cloudera-scm-agent/process type tmpfs (rw,mode=0751)

 

mount on host 2:

 

proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/xvdb on /mnt type ext3 (rw)
cm_processes on /run/cloudera-scm-agent/process type tmpfs (rw,mode=0751)

Re: Namenode data directory size problem

Cloudera Manager is complaining about /mnt/dfs/nn having less tha n5.0GB whereas /dfs/nn has more than than 5.0GB free.


/dev/xvdb 4.0G 73M 3.7G 2% /mnt

 

Please check your HDFS configuration for the edits directory. Just visit the HDFS service, open the configuration and search for /mnt. You should see where it is being used.

Regards,
Gautam Gopalakrishnan

Re: Namenode data directory size problem

New Contributor

Looking at the HDFS configuration, there are two directories listed for NameNode data:

 

/dfs/nn

/mnt/dfs/nn

 

This was setup by default by the installer. What do I need to do to fix the critical error at this point?

 

Thanks for your help so far.

 

Jayme

Re: Namenode data directory size problem

New Contributor
P.S. I also see the /mnt directory being used in "DataNode Default Group" (/mnt/dfs/dn) and "DataNode Group 1" (/mnt/dfs/dn) again.

Re: Namenode data directory size problem

The installer generally adds two directories for the namenode edits
directory to ensure two copies exist. In your case /mnt has very little
space and this is triggering the warning. If this is a test cluster you're
fine with just one edits directory. So you can delete /mnt/dfs/nn and
/mnt/dfs/dn from the configuration and restart HDFS. Do keep in mind that
if the sole copy gets corrupted somehow, you'll lose all your contents.

Regards,
Gautam Gopalakrishnan
Don't have an account?
Coming from Hortonworks? Activate your account here