01-25-2017 08:19 AM
I'm using Impala 2.2.0, cdh 5.4.3.
There are times that I have queries that had been canceled by the user, but they're appear in the deamon web UI in "Queries in flight" tab.
These queries in EXCEPTION state, and when I look into their profile, I can see that the status is "Cancelled".
Sounds great, BUT - when I look deeper, and see the memory usage for the specific query, It looks like the cancel command didn't free the memory.
In case of massive queries, this issue harm concurrency, and the coordinator host become a bottleneck.
(As u can see in the attached graph).
02-01-2017 05:27 AM
I'm aware for the cancel button, but in production env, where you can have massive amount of queries in EXCEPTION mode, canceling queries manually is not the right solution.
I wonder if it can be fixed by configuration or script.
Anyway, it looks like a bug to me.
Have anyone experienced this issue?
If yes, I would like to hear what was your action items.
02-01-2017 07:46 AM
Can you tell us which application issues the Cancel command (JDBC, ODBC)? Can you also check the log files to see what's going on? Usually grepping the logs for the query id may lead to stuck fragments or similar.
02-01-2017 09:20 AM
yeap. I agree with you. we can't just kill those quries manually.
I think most of times quries in exception status and in filght quries from Hue.
Is there way to log slow quries like MySQL slow quries?
Is there way to set timeout for long running quries so that those quries can be killed?
02-02-2017 05:35 AM
This page has information on various timeout settings for Impala: http://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html
If you're using CM, then all queries should be logged and you can search for long running queries in CM. Without CM I don't know of any convenient helper, so you may have to search/filter the logfiles yourself.