Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Impala Partition Query running Slow

avatar
Explorer

So I was trying to partition my Impala table with the column 'file' which has 1500 distinct records. That means 1500 partitions. I first run a query like this to return the partition queries:

SELECT DISTINCT
  concat('insert into partitioned_table partition (year=',
    cast(year as string),', month=',cast(month as string),
    ') select c1, c2, c3 from raw_data where year=',
    cast(year as string),' and month=',cast(month as string),';') AS command
  FROM raw_data;

Then I got 1500 queries to run.

搜狗截图20170207102602.png 

Now there's one problem: Since each query might take 3 minutes to finish. 1500 queries could take several days. Which is a really long time. To save the time, I have already done some tuning: using COMPUTE STATS to get statics, convert table to Parquet. MY question is, Is there a way that can speed up this process? Like max up the executors just like Hive can do?

 

My Cluster:

3 nodes, each with 6 core cpu, 32GB RAM and 1.8T hard disk

 

Any thoughts would be helpful, Thank you!

1 ACCEPTED SOLUTION

avatar
Explorer
1 REPLY 1

avatar
Explorer

I found out the solution. here is the link