Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

bucket y delta in hive

avatar
Master Collaborator

Hi:

I have lunched 5 batches, one per month and in hive directories i can see that why??? inside de delta there are all the bucket, is corret???

4756-snip20160604-1.png

4757-snip20160604-2.png

1 ACCEPTED SOLUTION

avatar
Super Collaborator

both compactions are important. The should run minor compactions more frequently to combine delta files into fewer deltas. From the listing above major compaction ran and produced the base file. All the delta files in the listing should be garbage collected by the system. Could you provide output of "show locks" and "show transactions"? The only thing that should prevent deletion of these delta files is a query that is still running which the "show" commands should indicate.

View solution in original post

17 REPLIES 17

avatar
Super Guru

@Roberto Sancho

Yes, if you are using ACID operation then these delta file should create for each partition and inside it you will see the no. of buckets which got some change during ACID operation. These delta files should be merge with base records as per the compaction policies.

avatar
Master Collaborator

Hi:

and i can do that manually?? how?

avatar
Super Guru

@Roberto Sancho

If compaction is enabled then you won't need to do it manually but if it disabled then you can use "ALTER TABLE" statement. Please see below doc for info.

ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] COMPACT 'compaction_type';

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/Par...

avatar
Super Guru

@Roberto Sancho

Did you restarted the hive services after changing delta threshold to 5?.

Did you tried to do manual compaction? if yes then was there any transaction files inside those directories after that?

avatar
Master Collaborator

Yes i restarted all hive services, i didnt try do manually, ill wait more days to see if its working, if not ill do manually. any other suggeston? o parameter that is wrong on my picture??

avatar
Super Guru

Rest of the parameters seems fine, Are you seeing any error in metastore logs?

Did you notice any compaction or cleaner thread logs in hivemeta?

avatar
Super Guru

Nothing seems wrong in the logs, Did it triggered automatically? if not have you tried manual way?

avatar
Super Guru

Can you please try manual compaction and see if those delta files getting merged?

avatar
Super Guru

@Roberto Sancho

I remember one bug in compaction if data load done through dynamic partition then metastore doesn't trigger compactor for that table/partition. However it should be fixed in 2.4 but just wanted to check if this is the case?