I have lunched 5 batches, one per month and in hive directories i can see that why??? inside de delta there are all the bucket, is corret???
both compactions are important. The should run minor compactions more frequently to combine delta files into fewer deltas. From the listing above major compaction ran and produced the base file. All the delta files in the listing should be garbage collected by the system. Could you provide output of "show locks" and "show transactions"? The only thing that should prevent deletion of these delta files is a query that is still running which the "show" commands should indicate.
Yes, if you are using ACID operation then these delta file should create for each partition and inside it you will see the no. of buckets which got some change during ACID operation. These delta files should be merge with base records as per the compaction policies.
If compaction is enabled then you won't need to do it manually but if it disabled then you can use "ALTER TABLE" statement. Please see below doc for info.
ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])]
Did you restarted the hive services after changing delta threshold to 5?.
Did you tried to do manual compaction? if yes then was there any transaction files inside those directories after that?
Yes i restarted all hive services, i didnt try do manually, ill wait more days to see if its working, if not ill do manually. any other suggeston? o parameter that is wrong on my picture??
I remember one bug in compaction if data load done through dynamic partition then metastore doesn't trigger compactor for that table/partition. However it should be fixed in 2.4 but just wanted to check if this is the case?