question Re: How to run manual compaction on the "nested partitions”, I tried searching on hive docs but could not find any example. in Support Questions

How to run manual compaction on the "nested partitions”, I tried searching on hive docs but could not find any example.

amresh_kumar — Fri, 10 Jun 2016 16:20:46 GMT

The nested partition structure is as below.

/dbdata/tblocation/ordersummery/tran_date=2016-04-06/ordertype=SUPE

hive> ALTER TABLE dtl.ordersummery PARTITION (tran_date='2016-04-06') COMPACT 'minor';

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Compaction can only be requested on one partition at a time.

hive> ALTER TABLE dtl.ordersummery PARTITION (tran_date='2016-04-06/ordertype=SUPE') COMPACT 'minor';

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Invalid partition spec specified

rajkumar_singh — Fri, 10 Jun 2016 16:23:38 GMT

can you try this?

ALTER TABLE dtl.ordersummery PARTITION (tran_date='2016-04-06',ordertype='SUPE') COMPACT 'minor';

amresh_kumar — Fri, 10 Jun 2016 16:47:03 GMT

thanks it works.

cgauthier — Sat, 11 Jun 2016 01:12:13 GMT

I'll add this to the HWKs hive docs. Thanks @Rajkumar Singh

srinivasp — Fri, 29 Sep 2017 00:00:11 GMT

CREATETABLE resource.hello_acid (key int, value int)

PARTITIONED BY (load_date date)

CLUSTERED BY(key) INTO 3 BUCKETS

STORED AS ORC TBLPROPERTIES ('transactional'='true');

INSERT INTO hello_acid partition (load_date='2016-03-03') VALUES (1, 1);

INSERT INTO hello_acid partition (load_date='2016-03-03') VALUES (2, 2);

INSERT INTO hello_acid partition (load_date='2016-03-03') VALUES (3, 3);

Everything look great at this point. I've been able to add rows, and query the table as usual.
This is the content of HDFS directory of table partition (/apps/hive/warehouse/resource.db/hello_acid/load_date=2016-03-03/): 3 delta directories (1 per insert transaction)

In Hive, I issued a minor compaction command. If i understood it right, this should have merged all delta directories into one. Didn't work!

ALTER TABLE hello_acid partition (load_date='2016-03-03') COMPACT 'minor';

Next, I issued a major compaction command. This should have deleted all delta files and created a base file, with all the info. Didn't work either!
Finally, I ran this last command:

SHOW COMPACTIONS;

+-----------+-------------------------------+-----------------------+--------+------------+---------------------------------+----------------+--+

+-----------+-------------------------------+-----------------------+--------+------------+---------------------------------+----------------+--+

+-----------+-------------------------------+-----------------------+--------+------------+---------------------------------+----------------+--+

gaurang_n_shah — Thu, 06 Sep 2018 00:54:04 GMT

you are not getting the desired result as your compaction has failed. please check the yarn log to understand what might have gone wrong.