28732
DISCUSSIONS
101745
MEMBERS
3157
ARTICLES
Created 01-18-2019 03:06 PM
I drop partitions for manage table then check the size of the table. I run below command to check.
hdfs du and hdfs fsck command shows different value of storage size
du t_table show 7TB
du t_table for all the partition of the table shows 600gb
fsck t_table show 600gb
Why it shows different values
hdfs dfs -du -s -h /data/warehouse/developer.db/t_table/
7.0 T 21.1 T /data/warehouse/developer.db/t_table
hdfs dfs -du -s -h /data/warehouse/developer.db/t_table/*
31.0 G 93.0 G /data/warehouse/developer.db/t_table/day_string=2019-01-01
30.9 G 92.7 G /data/warehouse/developer.db/t_table/day_string=2019-01-02
31.0 G 92.9 G /data/warehouse/developer.db/t_table/day_string=2019-01-03
31.0 G 93.0 G /data/warehouse/developer.db/t_table/day_string=2019-01-04
31.0 G 92.9 G /data/warehouse/developer.db/t_table/day_string=2019-01-05
31.1 G 93.2 G /data/warehouse/developer.db/t_table/day_string=2019-01-06
31.0 G 93.1 G /data/warehouse/developer.db/t_table/day_string=2019-01-07
31.0 G 93.1 G /data/warehouse/developer.db/t_table/day_string=2019-01-08
31.2 G 93.6 G /data/warehouse/developer.db/t_table/day_string=2019-01-09
31.0 G 93.1 G /data/warehouse/developer.db/t_table/day_string=2019-01-10
31.1 G 93.2 G /data/warehouse/developer.db/t_table/day_string=2019-01-11
31.1 G 93.4 G /data/warehouse/developer.db/t_table/day_string=2019-01-12
31.1 G 93.3 G /data/warehouse/developer.db/t_table/day_string=2019-01-13
31.1 G 93.4 G /data/warehouse/developer.db/t_table/day_string=2019-01-14
31.2 G 93.5 G /data/warehouse/developer.db/t_table/day_string=2019-01-15
31.1 G 93.2 G /data/warehouse/developer.db/t_table/day_string=2019-01-16
31.2 G 93.5 G /data/warehouse/developer.db/t_table/day_string=2019-01-17
31.1 G 93.4 G /data/warehouse/developer.db/t_table/day_string=2019-01-18
hdfs fsck /data/warehouse/developer.db/t_table/
Total size: 600360486250 B
Total dirs: 19
Total files: 1660