Skip to main content
Sunil1359
Bronze ⭐️⭐️⭐️
Sep 26, 2022 10:43 AM UTC
Question

Tez query profiling

Sunil1359_0-1664188587998.png

Sunil1359_1-1664188785677.png

 

Can someone explain why the compilation time is so big and also why we don't have other info in the dag timeline?
Also if you can explain what exactly happens when the dag is build, submitted and is running.

1 reply

ggangadharan
Gold ⭐️
Gold ⭐️
Oct 13, 2022 9:46 AM UTC

@Sunil1359  

Compilation might be higher if the table has a large number of partitions or if the HMS process is slow when the query runs.  Please check the below on the corresponding time period to find the root cause. 

HS2 log
HMS log
HMS jstack 

In Tez engine queries will run in the form of DAG.  In the compilation phase, once the semantic analysis process is completed, the plan will be generated depending on the query you submitted. explain <your query> gives the plan of the query. 

Once the plan is generated DAG will be submitted to yarn and the DAG will run depending on the plan. 

As part of DAG, Split generation, input file read, shuffle fetch ..etc will be taken care and the end result will be transferred to the client. 




Translate