Compute stats is an expensive operation For a table with below definition CREATE TABLE default.test123 ( a INT, b INT ) PARTITIONED BY ( c STRING ); When you run compute stats, it spawns child queries as follows SELECT NDV(a) AS a, CAST(-1 as BIGINT), 4, CAST(4 as DOUBLE), NDV(b) AS b, CAST(-1 as BIGINT), 4, CAST(4 as DOUBLE), NDV(c) AS c, CAST(-1 as BIGINT), MAX(length(c)), AVG(length(c)), NDV(d) AS d, CAST(-1 as BIGINT), 8, CAST(8 as DOUBLE), NDV(e) AS e, CAST(-1 as BIGINT), 8, CAST(8 as DOUBLE) FROM default.mytable i.e) compute stats does compute the distinct values in each columns[NDV - number of distinct values], max and average length values in string columns. This on a big table is an expensive operation and it takes more time and resource The profile of compute stats will contains the below section which will explain you the time taken for "Child queries" in nanoseconds Start execution: 0 Planning finished: 1999998 Child queries finished: 550999506 Metastore update finished: 847999239 Rows available: 847999239 Profile Collection: ================== a. Go to Impala > Queries b. Identify the query you are interested in and from the dropdown on the right select "Query Details" c. On the resulting page select "Download Profile" 1/ To understand the cpu utilisation that you highlighted here. Could you please provide the profile of the insert query. 2/ And how did you confirm that impala is causing 100% cpu utilisation? Did you run top and notice impalad process taking all the cpu?
... View more
