Explorer
Posts: 38
Registered: ‎09-29-2016
HDFS du and fsck command shows different storage value

I drop partitions for manage table then check the size of the table. I run below command to check.

hdfs du and hdfs fsck command shows different value of storage size

du t_table show 7TB

du t_table for all the partition of the table shows 600gb

fsck t_table show 600gb

 

Why it shows different values

 

hdfs dfs -du -s -h /data/warehouse/developer.db/t_table/
7.0 T 21.1 T /data/warehouse/developer.db/t_table

hdfs dfs -du -s -h /data/warehouse/developer.db/t_table/*
31.0 G 93.0 G /data/warehouse/developer.db/t_table/day_string=2019-01-01
30.9 G 92.7 G /data/warehouse/developer.db/t_table/day_string=2019-01-02
31.0 G 92.9 G /data/warehouse/developer.db/t_table/day_string=2019-01-03
31.0 G 93.0 G /data/warehouse/developer.db/t_table/day_string=2019-01-04
31.0 G 92.9 G /data/warehouse/developer.db/t_table/day_string=2019-01-05
31.1 G 93.2 G /data/warehouse/developer.db/t_table/day_string=2019-01-06
31.0 G 93.1 G /data/warehouse/developer.db/t_table/day_string=2019-01-07
31.0 G 93.1 G /data/warehouse/developer.db/t_table/day_string=2019-01-08
31.2 G 93.6 G /data/warehouse/developer.db/t_table/day_string=2019-01-09
31.0 G 93.1 G /data/warehouse/developer.db/t_table/day_string=2019-01-10
31.1 G 93.2 G /data/warehouse/developer.db/t_table/day_string=2019-01-11
31.1 G 93.4 G /data/warehouse/developer.db/t_table/day_string=2019-01-12
31.1 G 93.3 G /data/warehouse/developer.db/t_table/day_string=2019-01-13
31.1 G 93.4 G /data/warehouse/developer.db/t_table/day_string=2019-01-14
31.2 G 93.5 G /data/warehouse/developer.db/t_table/day_string=2019-01-15
31.1 G 93.2 G /data/warehouse/developer.db/t_table/day_string=2019-01-16
31.2 G 93.5 G /data/warehouse/developer.db/t_table/day_string=2019-01-17
31.1 G 93.4 G /data/warehouse/developer.db/t_table/day_string=2019-01-18

hdfs fsck /data/warehouse/developer.db/t_table/
Total size: 600360486250 B
Total dirs: 19
Total files: 1660

Who Me Too'd this topic