question Re: hdfs du format change in Archives of Support Questions (Read Only)

hdfs du format change

Mythobeast — Fri, 08 May 2015 15:27:03 GMT

I just switched from Cloudera 4 to Cloudera 5, and the output format for hdfs dfs -du has changed. It now has two columns instead of just one. I'm guessing that the first is the actual content size and the second is the block storage consumption, but I can't find any documentation about this. Can anyone clarify and/or point me the right direction?

Re: hdfs du format change

GautamG — Sun, 10 May 2015 23:45:49 GMT

My output looks like this on both CDH4 and CDH5.

# sudo -u hdfs hdfs dfs -du /
20143 /accumulo
3230 /hbase
0 /solr
4 /tmp
287725904 /user

Please paste your command and output here so we can have a look.

Re: hdfs du format change

Mythobeast — Mon, 11 May 2015 17:26:28 GMT

I'm on Cloudera 5.3.3. Here's my command line and output:

[hadoop]$ hdfs dfs -du /
2298676940886 6896030822658 /output
21297905593 63893716779 /tmp
6072184915396 18216555409976 /user

Re: hdfs du format change

GautamG — Wed, 13 May 2015 02:55:26 GMT

I checked on CDH 5.2 and it was the same as CDH4. The output you see was introduced in CDH 5.3, the jira being HADOOP-6857. The second column is supposed to calculate the raw disk usage of the file/directory.

So if a file is 42 bytes, it correctly shows (file size) * (replication factor of file)

$ hadoop fs -du /hbase/hbase.id
42 126 /hbase/hbase.id

However it seems incorrect for directories as column 2 shows (theoretical block size) * (replication factor of file) * (number of files in directory)

$ hadoop fs -du -h /hbase/WALs
0    0      /hbase/WALs/hregion-67213513
83   384 M /hbase/WALs/server54-2.abc.cloudera.com,22101,1431434548533
166 768 M /hbase/WALs/server54-3.abc.cloudera.com,22101,1431434548804
83   384 M /hbase/WALs/server54-4.abc.cloudera.com,22101,1431434547067

$ hadoop fs -ls -R /hbase/WALs/server54-2.abc.cloudera.com,22101,1431434548533
-rw-r--r--   3 hbase hbase         83 2015-05-12 18:42 /hbase/WALs/server54-2.abc.cloudera.com,22101,1431434548533/server54-2.abc.cloudera.com%2C22101%2C1431434548533.default.1431481363257

$ hadoop fs -ls -R /hbase/WALs/server54-3.abc.cloudera.com,22101,1431434548804
-rw-r--r--   3 hbase hbase         83 2015-05-12 19:42 /hbase/WALs/server54-3.abc.cloudera.com,22101,1431434548804/server54-3.abc.cloudera.com%2C22101%2C1431434548804..meta.1431484974652.meta
-rw-r--r--   3 hbase hbase         83 2015-05-12 19:42 /hbase/WALs/server54-3.abc.cloudera.com,22101,1431434548804/server54-3.abc.cloudera.com%2C22101%2C1431434548804.default.1431484963607

Re: hdfs du format change

GautamG — Wed, 13 May 2015 02:57:51 GMT

The link to the source is here

https://github.com/cloudera/hadoop-common/blob/cdh5.4.0-release/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java#L120