Reply
Highlighted
Explorer
Posts: 13
Registered: ‎08-07-2017

Understanding Impala operator time

Hello,

 

I ran an Impala query and tried to understand some time metrics for operators. In the exe summary, I have following:

 

    ExecSummary: 
Operator              #Hosts   Avg Time   Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                         
----------------------------------------------------------------------------------------------------------------------------------
....
06:HASH JOIN               1    1s051ms    1s051ms  857.79K          -1    8.66 MB      266.80 KB  INNER JOIN, BROADCAST          
|--11:EXCHANGE             1   11.914us   11.914us       61       7.30K          0              0  BROADCAST                      
|  03:SCAN HDFS            1   83.745ms   83.745ms       61       7.30K   18.05 MB       48.00 MB  tpcds_text_100.date_dim        
...

However, when I look into the fragment which has the Operator information, I see the following:

 

HASH_JOIN_NODE (id=6):(Total: 2m42s, non-child: 1s051ms, % non-child: 0.65%)

 

My question is, how could the total time of that operator being 2m42s while the avg/max time of the operator being only 1s51ms?

 

For complete Exec summary of this query, please see below:

 

    ExecSummary: 
Operator              #Hosts   Avg Time   Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                         
----------------------------------------------------------------------------------------------------------------------------------
14:MERGING-EXCHANGE        1   46.195us   46.195us      100         100          0        -1.00 B  UNPARTITIONED                  
08:TOP-N                   1    2.566ms    2.566ms      100         100   36.00 KB        6.15 KB                                 
13:AGGREGATE               1   60.442ms   60.442ms   54.98K          -1   14.26 MB      128.00 MB  FINALIZE                       
12:EXCHANGE                1    8.420ms    8.420ms   95.31K          -1          0              0  HASH(w_warehouse_name,i_ite... 
07:AGGREGATE               1  316.838ms  316.838ms   95.31K          -1   19.12 MB      128.00 MB  STREAMING                      
06:HASH JOIN               1    1s051ms    1s051ms  857.79K          -1    8.66 MB      266.80 KB  INNER JOIN, BROADCAST          
|--11:EXCHANGE             1   11.914us   11.914us       61       7.30K          0              0  BROADCAST                      
|  03:SCAN HDFS            1   83.745ms   83.745ms       61       7.30K   18.05 MB       48.00 MB  tpcds_text_100.date_dim        
05:HASH JOIN               1    3s907ms    3s907ms   25.08M          -1    8.40 MB        2.00 GB  INNER JOIN, BROADCAST          
|--10:EXCHANGE             1   11.369us   11.369us       15          -1          0              0  BROADCAST                      
|  01:SCAN HDFS            1   40.714ms   40.714ms       15          -1   41.00 KB       32.00 MB  tpcds_text_100.warehouse       
04:HASH JOIN               1      1m53s      1m53s   25.08M          -1   22.40 GB        2.00 GB  INNER JOIN, BROADCAST          
|--09:EXCHANGE             1   12s011ms   12s011ms  399.33M          -1          0              0  BROADCAST                      
|  00:SCAN HDFS            3    1s583ms    1s608ms  399.33M          -1  514.87 MB        3.44 GB  tpcds_text_100.inventory       
02:SCAN HDFS               1   43.921ms   43.921ms   12.86K      20.40K   64.67 MB      128.00 MB  tpcds_text_100.item 

 

Thanks,

S.

 

 

Announcements