Created 07-31-2017 09:20 AM
The HDFS Quota Guide, http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html, shows how to list details of quotas at a specific directory where the quota is listed, but is there a way to see all quotas with one command (or at least a way to list all directories that have quotas, something like the way you can list all snapshottable dirs, which I could then programmatically iterate through and check individual quotas?
My "hunch" was that I could just check on the / directory and see a roll-up of the two specific quotas showed first, but as expected it is only showing the details of that dir's quota (if it exist).
[hdfs@node1 ~]$ hdfs dfs -count -v -q /user/testeng QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 400 399 none inf 1 0 0 /user/testeng [hdfs@node1 ~]$ hdfs dfs -count -v -q /user/testmar QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME none inf 134352500 134352500 1 0 0 /user/testmar [hdfs@node1 ~]$ [hdfs@node1 ~]$ [hdfs@node1 ~]$ hdfs dfs -count -v -q / QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 9223372036854775807 9223372036854775735 none inf 49 23 457221101 / [hdfs@node1 ~]$
Created 07-31-2017 11:25 AM
Not sure whether we have a single command to get the quotas for all the directory. But I would try to get all the HDFS directories and iterate it through a shell script which get the directories list from HDFS and append it in a file or we could we even print it on the screen also.
Created 07-31-2017 11:29 AM
hadoop fs -ls -R / would get the list of directories and its sub directories. Save it in a file and read it line by line using shell commands and pass it as a variable to hadoop fs -count -v -q $linefrompreviouscommand. This would work.
Created 07-31-2017 12:34 PM
Yep, this could work, but for a big cluster I could imagine this being time-consuming. The initial recursive listing (especially since it will represent down to the file level) could be quite large for any file system of any size. The more time-consuming effort would be to run the "hdfs dfs -count" command over and over and over. But... like you said, this should work. Preferably, I'd want the NN to just offer a "show me all quoto details" or at least just "show me directories w/quotas". Since this function is not present, Maybe there is a performance hit for NN to quickly determine this that I'm not considering as seems lightweight to me. Thanks for your suggestion.