Tez query profiling

Sunil1359 — Mon, 26 Sep 2022 10:43:46 GMT

Can someone explain why the compilation time is so big and also why we don't have other info in the dag timeline?
Also if you can explain what exactly happens when the dag is build, submitted and is running.

Re: Tez query profiling

ggangadharan — Thu, 13 Oct 2022 09:46:09 GMT

@Sunil1359

Compilation might be higher if the table has a large number of partitions or if the HMS process is slow when the query runs. Please check the below on the corresponding time period to find the root cause.

HS2 log
HMS log
HMS jstack

In Tez engine queries will run in the form of DAG. In the compilation phase, once the semantic analysis process is completed, the plan will be generated depending on the query you submitted. explain <your query> gives the plan of the query.

Once the plan is generated DAG will be submitted to yarn and the DAG will run depending on the plan.

As part of DAG, Split generation, input file read, shuffle fetch ..etc will be taken care and the end result will be transferred to the client.

question Tez query profiling in Support Questions

Tez query profiling

Re: Tez query profiling