I find out that the Impala Daemon UI (default port 25000) has memz tab which shows impala query memory breakdown during query runtime. However, there seems to be some delay on the section which shows the "real-time" query information. In my tests, I find the Breakdown information shows up roughly after fragments start. Is this observation correct?
Moreover, is the data showing up in the section time accurate (reflect the actual mem consumption at that timepoint)? How often is the data on the webpage updated? For example, if I'm refreshing the page every 0.5 second, am I getting accurate memory consumption information every 0.5 second?
Thanks a lot,
The /memz information reflects that Impala daemon's view of memory consumption at the time you refresh the page, so there's no delay. There is likely some delay between the query starting and the query's fragments showing up on the page because the query memory tracker is only registered when the fragment starts up - i.e. after the query is planned,the fragment has been sent out, and it has started executing.