Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can you list ALL of HDFS quotas with a single command?

avatar

The HDFS Quota Guide, http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html, shows how to list details of quotas at a specific directory where the quota is listed, but is there a way to see all quotas with one command (or at least a way to list all directories that have quotas, something like the way you can list all snapshottable dirs, which I could then programmatically iterate through and check individual quotas?

My "hunch" was that I could just check on the / directory and see a roll-up of the two specific quotas showed first, but as expected it is only showing the details of that dir's quota (if it exist).

[hdfs@node1 ~]$ hdfs dfs -count -v -q /user/testeng
       QUOTA       REM_QUOTA     SPACE_QUOTA REM_SPACE_QUOTA    DIR_COUNT   FILE_COUNT       CONTENT_SIZE PATHNAME
         400             399            none             inf            1            0                  0 /user/testeng
[hdfs@node1 ~]$ hdfs dfs -count -v -q /user/testmar
       QUOTA       REM_QUOTA     SPACE_QUOTA REM_SPACE_QUOTA    DIR_COUNT   FILE_COUNT       CONTENT_SIZE PATHNAME
        none             inf       134352500       134352500            1            0                  0 /user/testmar
[hdfs@node1 ~]$ 
[hdfs@node1 ~]$ 
[hdfs@node1 ~]$ hdfs dfs -count -v -q /            
       QUOTA       REM_QUOTA     SPACE_QUOTA REM_SPACE_QUOTA    DIR_COUNT   FILE_COUNT       CONTENT_SIZE PATHNAME
9223372036854775807 9223372036854775735            none             inf           49           23          457221101 /
[hdfs@node1 ~]$ 
3 REPLIES 3

avatar

Hi @Lester Martin

Not sure whether we have a single command to get the quotas for all the directory. But I would try to get all the HDFS directories and iterate it through a shell script which get the directories list from HDFS and append it in a file or we could we even print it on the screen also.

avatar

hadoop fs -ls -R / would get the list of directories and its sub directories. Save it in a file and read it line by line using shell commands and pass it as a variable to hadoop fs -count -v -q $linefrompreviouscommand. This would work.

avatar

Yep, this could work, but for a big cluster I could imagine this being time-consuming. The initial recursive listing (especially since it will represent down to the file level) could be quite large for any file system of any size. The more time-consuming effort would be to run the "hdfs dfs -count" command over and over and over. But... like you said, this should work. Preferably, I'd want the NN to just offer a "show me all quoto details" or at least just "show me directories w/quotas". Since this function is not present, Maybe there is a performance hit for NN to quickly determine this that I'm not considering as seems lightweight to me. Thanks for your suggestion.