We recommend utilizing CDW for Kubernetes on Hive. Based on the description, it seems that you are currently using the apache-hive library. In the upstream (Apache), images have already been pushed to Docker Hub, so you can utilize the same. I have attached the relevant documents for your reference. https://hive.apache.org/development/quickstart/ https://docs.cloudera.com/data-warehouse/cloud/overview/topics/dw-service-architecture.html ... View more

@Lorenzo The issue seems to be related to HIVE-27191 where some mhl_txnids do not exist in TXNS,completed_txn_components txn_components table but they are still present in min_history_level table, as a result, the cleaner gets blocked and many entries are stuck in the ready-for-cleaning state. To confirm that collect the output of below query   SELECT MHL_TXNID FROM HIVE.MIN_HISTORY_LEVEL WHERE MHL_MIN_OPEN_TXNID = (SELECT MIN(MHL_MIN_OPEN_TXNID) FROM HIVE.MIN_HISTORY_LEVEL);   Once we get the output of the above query check if those txn ids are there in TXNS,completed_txn_components txn_components tables using below commands.   select * from txn_components where tc_txnid IN (MHL_TXNID ); select * from completed_txn_components where ctc_txnid IN (MHL_TXNID); select * from TXNS where ctc_txnid IN (MHL_TXNID);   If we got 0 results from the above queries this confirms that the MHL_TXNIDs we got above are orphans and we need to remove them in order to unblock the cleaner.   delete from MIN_HISTORY_LEVEL where MHL_TXNID=13422; --(repeat for all)   Hope this helps you in resolving the issue ... View more

If setting the proper queue name resolves the problem, it is possible that the query may have been submitted in the default queue, where it competes for resources with other queries and fails due to a timeout error ... View more

When writing to a statically partitioned table using HWC, the following query is internally fired to Hive through JDBC after writing data to a temporary location: Spark write statement: df.write.format(HIVE_WAREHOUSE_CONNECTOR).mode("append").option("partition", "c1='val1',c2='val2'").option("table", "t1").save();  HWC internal query: LOAD DATA INPATH '<spark.datasource.hive.warehouse.load.staging.dir>' [OVERWRITE] INTO TABLE db.t1 PARTITION (c1='val1',c2='val2');  During static partitioning, the partition information is known during compile time, resulting in the creation of a staging directory in the partition directory.   On the other hand, when writing to a dynamically partitioned table using HWC, the following query is internally fired to Hive through JDBC after writing data to a temporary location: Spark write statement: df.write.format(HIVE_WAREHOUSE_CONNECTOR).mode("append").option("partition", "c1='val1',c2").option("table", "t1").save(); HWC internal query: CREATE TEMPORARY EXTERNAL TABLE db.job_id_table(cols....) STORED AS ORC LOCATION '<spark.datasource.hive.warehouse.load.staging.dir>'; INSERT INTO TABLE t1 PARTITION (c1='val1',c2) SELECT <cols> FROM db.job_id_table; During dynamic partitioning, the partition information is known during runtime, hence the staging directory is created at the table level. Once the DAG  is completed, the MOVE TASK will move the files to the respective partitions.  ... View more