We are using CDH5.8.4, We are facing the issue with the storage under '/' directory is reaching 100%. But we are unable to figer out whuch directory/file causing the issue. Because all other mount points also point from '/' directory.
If any one know on this issue or any one have faces similar experience please let me know.
This is more of Unix question rathen than hadoop 🙂
Please try below command which will ignore all mounts and only give size details of filesystem/dirs which resides under "/"
for a in /*; do mountpoint -q -- "$a" || du -s -h -x "$a"; done
Let me know if that helps.
Hi @sagarshimpi , Thanks for your help.
We found that one of the directory is not mounted and stated using under '/' directory storage. Now the chalenge is how to mount the directory to a device with out losing the data.
Is there any way to add mount point to that directory?
Hi @Ba The only way is to copy/move the data to new mount point and then rename.
There is no short steps to faster the operation.
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Hi @BaluD Below will be the steps
1. Lets assume your exsiting data resides in /test [which is wrong mounted on "/"]
2. Create new mount point. eg /data01 [which is /dev/vdd1]
3. Mount the disk [which is wrong mount as off now] on the new mount created in steps1 ie. /data01
4. cp data from existing disk /test to /data01
5. Once copied, test data exist and then remove /test
6. unmount /data01
7. Create mount with /test
8. mount disk to new mount - mount /dev/vdd1 on /test
Hope steps are clear now.
@sagarshimpi Thanks for detailed steps, For the clarification will the namenode detects as same blocks with out loosing them or will result any files corrupted?
Note: There might be one replicated files
If this is namenode mount point then make sure the data is backed properly before you move/delete the filesystem.
There will be no impact unless the mount point name is same. Just make sure while you do this activity the namenode was not functional to client. It should be stopped.
I see there is other option -
You can better add the new mount point in HDFS configs ->dfs.name.data.dir [as comma seperated value]
Once you change the config and restart Namenode will start writing data to new mount. Once you see that all data is written to new mount comparing size with existing mount point then you can remove old mount from dfs.name.data.dir and restart hdfs
Make sure you do below first -
1. hdfs dfsadmin -safemode on
2. hdfs dfsadmin -saveNamespace
@sagarshimpi Thanks for explanation, But here the problem with HDFS one of data directory on that host.
What prerequisites need to take befor doing the stpes you informed before, like stop NN/DN or keep in NN in safe mode...?
Is HDFS block location points data directory name? Not depends on the device?
Will it workout even the directory has blocks of one replication factor files.
Hi @Ba As mentioned earlier,
If you are performing copying and remounting of Filesystem then during this activity namenode must be in safe mode and not serving any kind of operation.
If you add new mount to "dfs.name.data.dir", there there is no need to perform any steps only restart of hdfs will be required.
No activity for datanodes.
@sagarshimpi For example if I have 5 DataNodes(DNs) and each DN has 10(Disks) DataDirectories individual mount points(Like /Disk1,/Disk2,......,/Disk10)
How to list If I want the list of which files blocks are stored in DN1/Disk2
Hi @Ba You can probably fetch this information from hdfs cli rather than going to individual datanode disk.
you can run -
hdfs fsck / -files -blocks -locations -->This will give you entire files blocks information with locations on which datanodes the blocks are stored. Just to segregatate wrt each datanodes you can apply some "grep" or "awk" filter
Hope this is what you were looking for.
My understanding is "the above command can give information with locations on which datanodes the blocks are stored."
But My question is that can it give the /Disk1 or /Disk2 level information also?
I am not sure if following approach will solve your issue.
But the following article suggest that we can create and deploy a custom Alert Script to get more detailed information about the Disk usage.
Which Disk is consuming how much space may be you can enhance the logic a bit more to meet your exact requirement.