Debugging query in Impala

[#1]

Hi All,

We have an Impala query we want to investigate,

the query is stuck and I don't see it on the Queries tab on Impala service in CM.

Is there an option to debug a query,

some command I can run on the server before the execution of the query and see where it get stucks and get more info about it?

Thanks

[#2]

How are you submitting your query?

If through impala-shell then you should see something like  "Query progress can be monitored at: http://[coordinator_hostname]:25000/query_plan?query_id=984ed18511f4ae82:9ccc11c300000000 " and you could go there to see its progress.  Or you can start impala-shell with --live_summary and see progress of each fragment in realtime.

If through odbc/jdbc, and you're specifying a node directly (not through haproxy), then you can go directly here for that node:

http://[coordinator_hostname]:25000/queries

and see any queries running there, even if for some reason they're not coming up in CM.

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/impala_live_summary.html#live_summary

[#3]

Hi,

thanks for the reply.

But my problem is that I can't access "http://[coordinator_hostname]:25000" when I'm running the query,

somehow the URL is frozen.

even when I'm running it on Impala-shell with live-summary I don't get to the state when it spills tracing output cause it get stuck before doing it.

so I wanted to know if there is an option to trace the query right after I execute it on impala-shell and see on which stage it get stuck.

Thanks for the help

[#4]

Well if you can't access the impala UI on that node then you have bigger problems than that query.  Perhaps your impalad is hung? or maybe you have a firewall or network policy that is not allowing you to access that port?

Could you first of all try restarting that impalad?

[#5]

Hi,

I don't have a firewall because before running the query I can work with the daemon.

Yes, the daemon is hung every time I start executing the query and I want to check on what,

but I don't find anything so I wanted to debug the query and see step after step on what phase it stuck.

After restarting ImpalaD the daemon is working again until I issue the query again.

Thanks

[#6]

So this query hangs the daemon but other queries run fine?  Oh wow that's strange, I would expect it to fail with out-of-memory error or something.

Do you have really low memory on the daemon, like under 16GB?

You should review the daemon INFO log under /var/log/impalad and search for the query id, see how it progresses and where it gets stuck (you can compare entries vs. another query that runs fine).  You'll see a lot of info about fragments being setup and distributed.

[#7]

It's stuck in a CREATED state (if I remember correctly at one time I could get to the daemon page), I don't remember about other queries cause when I've checked only this query was on the daemon and the daemon was page stuck in my browser.

My daemon has exactly 16GB.

Thanks, I'll try that, there any other tracing tool I can use to check the daemon or the query?