We've come to strange situation with transactional tables. For several dropped tables we've see many compactions with "attempted" state. There is no files, no tables, but compactions still in compaction queue. As i understand this compactions never will be completed. Is there any way to remove this compactions from queue?
Will be pleased any help.
P.S. Sorry for my english.
Attempted means that the initiator attempted to schedule a compaction but it failed. As such, there will be no compaction associated with those requests. Other than visually unpleasant, there is no reason to worry about. They are part of the metastore log and it will show upto the threshold set. See threshold for attempted below. They will not be displayed after that number.
|hive.compactor.history.retention.succeeded||Default: 3||Metastore||Number of successful compaction entries to retain in history (per partition).|
|hive.compactor.history.retention.failed||Default: 3||Metastore||Number of failed compaction entries to retain in history (per partition).|
|hive.compactor.history.retention.attempted||Default: 2||Metastore||Number of attempted compaction entries to retain in history (per partition).|
|hive.compactor.initiator.failed.compacts.threshold||Default: 2||Metastore||Number of of consecutive failed compactions for a given partition after which the Initiator will stop attempting to schedule compactions automatically. It is still possible to use ALTER TABLE to initiate compaction. Once a manually initiated compaction succeeds auto initiated compactions will resume. Note that this must be less than hive.compactor.history.retention.failed.|
|hive.compactor.history.reaper.interval||Default: 2m||Metastore||Controls how often the process to purge historical record of compactions runs.|
If this was helpful, please vote/accept best answer.
A little bit of theory below for others who may have a similar question.
Compactions are initiated automatically, but can also be initiated manually with an ALTER TABLE COMPACT statement.
This may be due to not having https://issues.apache.org/jira/browse/HIVE-10632 in your build. There is some data in internal tables that was not cleaned when tables were dropped which is causing Initiator to try to schedule compactions.