Created on
01-27-2021
10:23 PM
- last edited on
01-27-2021
10:50 PM
by
VidyaSargur
Env:
HDP 3.1.5
Hive: hive_3_1_5_0_152
We have a hive table with 6k partitions, it is very normal when describing a single partition but super slow when describing the whole table, describing non-partitioned tables is normal.
desc hive_desc_partition_tbl_01; -- cost 1k+ sec
desc hive_desc_partition_tbl_02 partition(ns='NS1',slicetime=20200102); -- cost 3 sec
alter table hive_desc_partition_tbl_01 change column id id string comment 'change'; -- cost 3600 sec
The DDL of the table:
create TABLE `hive_desc_partition_tbl_01` (
`id` bigint,
`b` string,
`c` string,
`d` string,
`e` string,
`ns_name` string)
COMMENT 'compress test'
PARTITIONED BY (
`ns` string,
`slicetime` string)
Any ideas to optimize? Thanks in advance!
Created 01-29-2021 03:01 PM
I suspect the cause for "alter table" slowness is same as described here
HIVE-23959 & HIVE-23806 might help your situation.