Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

HDFS quota: Is there a GUI to control HDFS Name and User quota?

 
1 ACCEPTED SOLUTION

Accepted Solutions

Mentor

@Neeraj Sabharwal

AFAIK, there's no quotas management view in Ambari, I wrote a short tutorial on quotas,

### QUOTAS ###
# quotas can be number of files or size per directory, independent of each other
sudo -u hdfs hdfs dfs -mkdir /quotasdir
# requires superuser privileges
# set space quota of 1kb on a directory, can be k, m, g, etc.
sudo -u hdfs hdfs dfsadmin -setSpaceQuota 1k /quotasdir
# add a file
sudo -u hdfs hdfs dfs -touchz /quotasdir/1
# notice file is 0 bytes
sudo -u hdfs hdfs dfs -ls /quotasdir/
# for demo purposes, we need to upload a large file, larger than 1kb into directory, watch the prompt
sudo -u hdfs hdfs dfs -chown -R root:hdfs /quotasdir
hdfs dfs -put /root/install.log /quotasdir/
15/11/25 15:10:47 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /quotasdir is exceeded: quota = 1024 B = 1 KB but diskspace consumed = 402653184 B = 384 MB
at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:211)
at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:239)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:907)
# remove space quota
sudo -u hdfs hdfs dfsadmin -clrSpaceQuota /quotasdir
# try uploading again
hdfs dfs -put /root/install.log /quotasdir/
# ls on the directory
sudo -u hdfs hdfs dfs -ls /quotasdir/
# set name quota on directory, specifies how many files or directories per quota directory where # 2 is number of files or directories allowed in the quota dir
sudo -u hdfs hdfs dfsadmin -setQuota 3 /quotasdir
# add file, nothing happens
sudo -u hdfs hdfs dfs -touchz /quotasdir/1
# add another file and watch the prompt
sudo -u hdfs hdfs dfs -touchz /quotasdir/2
touchz: The NameSpace quota (directories and files) of directory /quotasdir is exceeded: quota=2 file count=3
# remove name quota
sudo -u hdfs hdfs dfsadmin -clrQuota /quotasdir
# now adding another file works
sudo -u hdfs hdfs dfs -touchz /quotasdir/2
# ls on the directory
sudo -u hdfs hdfs dfs -ls /quotasdir/
# An an extension to the count command of the HDFS shell reports quota values and the current count of names and bytes in use.
fs -count -q <directory>...<directory>With the -q option, also report the name quota value set for each directory, the available name quota remaining, the space quota value set, and the available space quota remaining. If the directory does not have a quota set, the reported values are none and inf.# make sure all quotas are cleared
sudo -u hdfs hdfs dfsadmin -clrSpaceQuota /quotasdir
sudo -u hdfs hdfs dfsadmin -clrQuota /quotasdir
sudo -u hdfs hdfs dfs -count -q /quotasdir
 none  inf  none  inf  1  4  9447 /quotasdir
# set quotas and check again
sudo -u hdfs hdfs dfsadmin -setSpaceQuota 99g /quotasdir
sudo -u hdfs hdfs dfsadmin -setQuota 8888 /quotasdir
8888  8883  106300440576  106300412235  1  4  9447 /quotasdir
sudo -u hdfs hdfs dfs -count -q /quotasdir
# 8888 is number of files
# 106300440576 is 99GB

View solution in original post

3 REPLIES 3

Mentor

@Neeraj Sabharwal

AFAIK, there's no quotas management view in Ambari, I wrote a short tutorial on quotas,

### QUOTAS ###
# quotas can be number of files or size per directory, independent of each other
sudo -u hdfs hdfs dfs -mkdir /quotasdir
# requires superuser privileges
# set space quota of 1kb on a directory, can be k, m, g, etc.
sudo -u hdfs hdfs dfsadmin -setSpaceQuota 1k /quotasdir
# add a file
sudo -u hdfs hdfs dfs -touchz /quotasdir/1
# notice file is 0 bytes
sudo -u hdfs hdfs dfs -ls /quotasdir/
# for demo purposes, we need to upload a large file, larger than 1kb into directory, watch the prompt
sudo -u hdfs hdfs dfs -chown -R root:hdfs /quotasdir
hdfs dfs -put /root/install.log /quotasdir/
15/11/25 15:10:47 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /quotasdir is exceeded: quota = 1024 B = 1 KB but diskspace consumed = 402653184 B = 384 MB
at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:211)
at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:239)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:907)
# remove space quota
sudo -u hdfs hdfs dfsadmin -clrSpaceQuota /quotasdir
# try uploading again
hdfs dfs -put /root/install.log /quotasdir/
# ls on the directory
sudo -u hdfs hdfs dfs -ls /quotasdir/
# set name quota on directory, specifies how many files or directories per quota directory where # 2 is number of files or directories allowed in the quota dir
sudo -u hdfs hdfs dfsadmin -setQuota 3 /quotasdir
# add file, nothing happens
sudo -u hdfs hdfs dfs -touchz /quotasdir/1
# add another file and watch the prompt
sudo -u hdfs hdfs dfs -touchz /quotasdir/2
touchz: The NameSpace quota (directories and files) of directory /quotasdir is exceeded: quota=2 file count=3
# remove name quota
sudo -u hdfs hdfs dfsadmin -clrQuota /quotasdir
# now adding another file works
sudo -u hdfs hdfs dfs -touchz /quotasdir/2
# ls on the directory
sudo -u hdfs hdfs dfs -ls /quotasdir/
# An an extension to the count command of the HDFS shell reports quota values and the current count of names and bytes in use.
fs -count -q <directory>...<directory>With the -q option, also report the name quota value set for each directory, the available name quota remaining, the space quota value set, and the available space quota remaining. If the directory does not have a quota set, the reported values are none and inf.# make sure all quotas are cleared
sudo -u hdfs hdfs dfsadmin -clrSpaceQuota /quotasdir
sudo -u hdfs hdfs dfsadmin -clrQuota /quotasdir
sudo -u hdfs hdfs dfs -count -q /quotasdir
 none  inf  none  inf  1  4  9447 /quotasdir
# set quotas and check again
sudo -u hdfs hdfs dfsadmin -setSpaceQuota 99g /quotasdir
sudo -u hdfs hdfs dfsadmin -setQuota 8888 /quotasdir
8888  8883  106300440576  106300412235  1  4  9447 /quotasdir
sudo -u hdfs hdfs dfs -count -q /quotasdir
# 8888 is number of files
# 106300440576 is 99GB

View solution in original post

Mentor

and this? @Neeraj Sabharwal

@Artem Ervits Its wikified..great answer but not what exact and I believe there is no GUI. You should write an article based on your answer.