Impala Partition Query running Slow

Beebeegun — Fri, 16 Sep 2022 11:01:57 GMT

So I was trying to partition my Impala table with the column 'file' which has 1500 distinct records. That means 1500 partitions. I first run a query like this to return the partition queries:

SELECT DISTINCT
  concat('insert into partitioned_table partition (year=',
    cast(year as string),', month=',cast(month as string),
    ') select c1, c2, c3 from raw_data where year=',
    cast(year as string),' and month=',cast(month as string),';') AS command
  FROM raw_data;

Then I got 1500 queries to run.

Now there's one problem: Since each query might take 3 minutes to finish. 1500 queries could take several days. Which is a really long time. To save the time, I have already done some tuning: using COMPUTE STATS to get statics, convert table to Parquet. MY question is, Is there a way that can speed up this process? Like max up the executors just like Hive can do?

My Cluster:

3 nodes, each with 6 core cpu, 32GB RAM and 1.8T hard disk

Any thoughts would be helpful, Thank you!

Re: Impala Partition Query running Slow

Beebeegun — Mon, 13 Feb 2017 15:27:10 GMT

I found out the solution. here is the link

question Re: Impala Partition Query running Slow in Archives of Support Questions (Read Only)

Impala Partition Query running Slow

Re: Impala Partition Query running Slow