Support Questions

Find answers, ask questions, and share your expertise

Worker node '/' directory is full Alerts

New Contributor

Hello Members/Team

 

We are using CDH5.8.4, We are facing the issue with the storage under '/' directory is reaching 100%. But we are unable to figer out whuch directory/file causing the issue. Because all other mount points also point from '/' directory.

edl25_n.png

 

 

If any one know on this issue or any one have faces similar experience please let me know.

 

Thanks

Balu

18 REPLIES 18

Expert Contributor

Hi @BaluD 

 

This is more of Unix question rathen than hadoop 🙂

 

Please try below command which will ignore all mounts and only give size details of filesystem/dirs which resides under "/"

 

for a in /*; do mountpoint -q -- "$a" || du -s -h -x "$a"; done

 Let me know if that helps.

Explorer

Hi @sagarshimpi , Thanks for your help.

 

We found that one of the directory is not mounted and stated using under '/' directory storage. Now the chalenge is how to mount the directory to a device with out losing the data.

 

Is there any way to add mount point to that directory?

 

Thanks

Balu

Expert Contributor

Hi @Ba The only way is to copy/move the data to new mount point and then rename.

There is no short steps to faster the operation.

 

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

 

New Contributor

@sagarshimpi  What do you mean by copy and rename? Can you please elaborate the steps?

 

Thanks

Balu

Expert Contributor

Hi @BaluD  Below will be the steps

 

1. Lets assume your exsiting data resides in /test [which is wrong mounted on "/"]

2. Create new mount point. eg /data01 [which is /dev/vdd1]

3. Mount the disk [which is wrong mount as off now] on the new mount created in steps1 ie. /data01

4. cp data from existing disk /test to /data01

5. Once copied, test data exist and then remove /test

6. unmount /data01

7. Create mount with /test

8. mount disk to new mount  - mount /dev/vdd1 on /test

 

Hope steps are clear now.

 

New Contributor

@sagarshimpi  Thanks for detailed steps, For the clarification will the namenode detects as same blocks with out loosing them or will result any files corrupted?

 

Note: There might be one replicated files

 

Thanks

Balu

Expert Contributor

Hi @BaluD

 

If this is namenode mount point then make sure the data is backed properly before you move/delete the filesystem.

 

There will be no impact unless the mount point name is same. Just make sure while you do this activity the namenode was not functional to client. It should be stopped.

 

I see there is other option -

 

You can better add the new mount point in HDFS configs ->dfs.name.data.dir [as comma seperated value]

Once you change the config and restart Namenode will start writing data to new mount. Once you see that all data is written to new mount comparing size with existing mount point then you can remove old mount from dfs.name.data.dir and restart hdfs

 

Make sure you do below first -

1. hdfs dfsadmin -safemode on

2. hdfs dfsadmin -saveNamespace

Explorer

@sagarshimpi  Thanks for explanation, But here the problem with HDFS one of data directory on that host.

What prerequisites need to take befor doing the stpes you informed before, like stop NN/DN or keep in NN in safe mode...?

 

Is HDFS block location points data directory name? Not depends on the device?

 

Will it workout even the directory has blocks of one replication factor files.

 

Thanks

Balu

Expert Contributor

Hi @Ba As mentioned earlier,

 

If you are performing copying and remounting of Filesystem then during this activity namenode must be in safe mode and not serving any kind of operation.

 

If you add new mount to "dfs.name.data.dir", there there is no need to perform any steps only restart of hdfs will be required.

No activity for datanodes.

Explorer

@sagarshimpi  OK, Let us work on this and update you with the result.

Expert Contributor

Hi @Ba  Did you test ? Do you have any update?

 

 

Explorer

@sagarshimpi I'll update you once I do.

Thanks

Balu

Explorer

Hi @sagarshimpi

 

Is there any way to list the HDFS files under DataDirectory level?

Expert Contributor

@Ba  i didn't get you. can you please elaborate more.

Explorer

@sagarshimpi For example if I have 5 DataNodes(DNs) and each DN has 10(Disks) DataDirectories individual mount points(Like /Disk1,/Disk2,......,/Disk10)

 

How to list If I want the list of which files blocks are stored in DN1/Disk2

 

Thanks

Balu

Expert Contributor

Hi @Ba  You can probably fetch this information from hdfs cli  rather than going to individual datanode disk.

 

you can run -

hdfs fsck / -files -blocks -locations
-->This will give you entire files blocks information with locations on which datanodes the blocks are stored.

Just to segregatate wrt each datanodes you can apply some "grep" or "awk" filter

 

Hope this is what you were looking for.

Explorer

My understanding is "the above command can give information with locations on which datanodes the blocks are stored."

 

But My question is that can it give the /Disk1 or /Disk2 level information also?

 

Example block1-DN2-Disk2

 

Thanks

Balu

Super Mentor

@Ba 

I am not sure if following approach will solve your issue.

 

But the following article suggest that we can create and deploy a custom Alert Script to get more detailed information about the Disk usage.

 

Which Disk is consuming how much space may be you can enhance the logic a bit more to meet your exact requirement. 

https://community.cloudera.com/t5/Community-Articles/How-to-create-and-register-custom-ambari-alerts...

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.