Skip to main content
sujonict07
Silver ⭐️⭐️
June 1, 2021
Question

HIVE: No Cleanup working after Compaction:

I have run manually minor and major compaction. Table data is referring to the new delta and base folder but the old delta and base folders are not removing and it consumes the hive storage. Moreover, No running job is referring to this table.
Queries: 

 

create table compaction_check.check_file(name string, age int);
insert into compaction_check.check_file values('user 1', 10),('user 2', 20),('user 3', 30);
insert into compaction_check.check_file values('user 4', 10),('user 5', 30),('user 6', 15);
ALTER TABLE compaction_check.check_file COMPACT 'major';

 

 Check input files:

 

select distinct INPUT__FILE__NAME FROM compaction_check.check_file;

 

output: Screenshot 2021-06-01 at 20.19.12.png

 Actual table directory :
Screenshot 2021-06-01 at 20.18.29.png

 

Why stale the base and delta folders are not removed?

 



 

6 replies

aakulov
Platinum ⭐️
Platinum ⭐️
June 1, 2021

Hello,

 

Could you please provide the output of 

SHOW TBLPROPERTIES compaction_check.check_file;

Also, which version of CDH/HDP or CDP are you using?

 

 

Thank you,

Alex

Translate
sujonict07
Silver ⭐️⭐️
June 2, 2021

Hi @aakulov, 

I am using Cloudera 7.1.3 and HIVE version: 3.1.3

Table TBLPROPERTIES Output: 

Screenshot 2021-06-02 at 9.27.06 AM.png

 


Thanks in Advance. 

Translate
aakulov
Platinum ⭐️
Platinum ⭐️
June 2, 2021

Hello again,

 

Base on the output, looks like the table itself is using the default compaction settings. That's good. Next place to check is the global Hive parameters and their values. Note that some of these settings are turned off by default (e.g. hive.compactor.initiator.on = false). See documentation below:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/managing-hive/topics/hive-compact-properties.html

 

Hope this helps,

Alex

Translate