bucket y delta in hive

[#1]

Hi:

I have lunched 5 batches, one per month and in hive directories i can see that why??? inside de delta there are all the bucket, is corret???

[#2]

@Roberto Sancho

Yes, if you are using ACID operation then these delta file should create for each partition and inside it you will see the no. of buckets which got some change during ACID operation. These delta files should be merge with base records as per the compaction policies.

[#3]

Hi:

and i can do that manually?? how?

[#4]

@Roberto Sancho

If compaction is enabled then you won't need to do it manually but if it disabled then you can use "ALTER TABLE" statement. Please see below doc for info.

ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] COMPACT 'compaction_type';

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/Par...

[#5]

@Roberto Sancho

Did you restarted the hive services after changing delta threshold to 5?.

Did you tried to do manual compaction? if yes then was there any transaction files inside those directories after that?

[#6]

Yes i restarted all hive services, i didnt try do manually, ill wait more days to see if its working, if not ill do manually. any other suggeston? o parameter that is wrong on my picture??

[#7]

Rest of the parameters seems fine, Are you seeing any error in metastore logs?

Did you notice any compaction or cleaner thread logs in hivemeta?

[#8]

Nothing seems wrong in the logs, Did it triggered automatically? if not have you tried manual way?

[#9]

Can you please try manual compaction and see if those delta files getting merged?

[#10]

@Roberto Sancho

I remember one bug in compaction if data load done through dynamic partition then metastore doesn't trigger compactor for that table/partition. However it should be fixed in 2.4 but just wanted to check if this is the case?