Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

bucket y delta in hive

Solved Go to solution
Highlighted

bucket y delta in hive

Super Collaborator

Hi:

I have lunched 5 batches, one per month and in hive directories i can see that why??? inside de delta there are all the bucket, is corret???

4756-snip20160604-1.png

4757-snip20160604-2.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: bucket y delta in hive

Expert Contributor

both compactions are important. The should run minor compactions more frequently to combine delta files into fewer deltas. From the listing above major compaction ran and produced the base file. All the delta files in the listing should be garbage collected by the system. Could you provide output of "show locks" and "show transactions"? The only thing that should prevent deletion of these delta files is a query that is still running which the "show" commands should indicate.

17 REPLIES 17

Re: bucket y delta in hive

@Roberto Sancho

Yes, if you are using ACID operation then these delta file should create for each partition and inside it you will see the no. of buckets which got some change during ACID operation. These delta files should be merge with base records as per the compaction policies.

Re: bucket y delta in hive

Super Collaborator

Hi:

and i can do that manually?? how?

Re: bucket y delta in hive

@Roberto Sancho

If compaction is enabled then you won't need to do it manually but if it disabled then you can use "ALTER TABLE" statement. Please see below doc for info.

ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] COMPACT 'compaction_type';

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/Par...

Re: bucket y delta in hive

@Roberto Sancho

Did you restarted the hive services after changing delta threshold to 5?.

Did you tried to do manual compaction? if yes then was there any transaction files inside those directories after that?

Re: bucket y delta in hive

Super Collaborator

Yes i restarted all hive services, i didnt try do manually, ill wait more days to see if its working, if not ill do manually. any other suggeston? o parameter that is wrong on my picture??

Re: bucket y delta in hive

Rest of the parameters seems fine, Are you seeing any error in metastore logs?

Did you notice any compaction or cleaner thread logs in hivemeta?

Re: bucket y delta in hive

Nothing seems wrong in the logs, Did it triggered automatically? if not have you tried manual way?

Re: bucket y delta in hive

Can you please try manual compaction and see if those delta files getting merged?

Re: bucket y delta in hive

@Roberto Sancho

I remember one bug in compaction if data load done through dynamic partition then metastore doesn't trigger compactor for that table/partition. However it should be fixed in 2.4 but just wanted to check if this is the case?

Don't have an account?
Coming from Hortonworks? Activate your account here