queries in flight issue

[#1]

Hey all,

I'm using Impala 2.2.0, cdh 5.4.3.

There are times that I have queries that had been canceled by the user, but they're appear in the deamon web UI in "Queries in flight" tab.

These queries in EXCEPTION state, and when I look into their profile, I can see that the status is "Cancelled".

Sounds great, BUT - when I look deeper, and see the memory usage for the specific query, It looks like the cancel command didn't free the memory.

In case of massive queries, this issue harm concurrency, and the coordinator host become a bottleneck.

(As u can see in the attached graph).

Any ideas?

Thanks

 Dror

[#2]

You might need to click the `cancel` link next to query.

Although a query is under exception status, it seems still running and using resources

Gatsby

[#3]

Thanks @thewayofthinkin.
I'm aware for the cancel button, but in production env, where you can have massive amount of queries in EXCEPTION mode, canceling queries manually is not the right solution.

I wonder if it can be fixed by configuration or script.

Anyway, it looks like a bug to me.

Have anyone experienced this issue?

If yes, I would like to hear what was your action items.

Thanks

Dror

[#4]

Hi dsss,

Can you tell us which application issues the Cancel command (JDBC, ODBC)? Can you also check the log files to see what's going on? Usually grepping the logs for the query id may lead to stuck fragments or similar.

Cheers, Lars

[#5]

@dsss

yeap. I agree with you. we can't just kill those quries manually.

I think most of times quries in exception status and in filght quries from Hue.

@Lars Volker

Is there way to log slow quries like MySQL slow quries?

Is there way to set timeout for long running quries so that those quries can be killed?

Thank you

Gatsby

[#6]

@thewayofthinkin

This page has information on various timeout settings for Impala: http://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html

If you're using CM, then all queries should be logged and you can search for long running queries in CM. Without CM I don't know of any convenient helper, so you may have to search/filter the logfiles yourself.

[#7]

Thank you.

I will check with team to set the timeout.

Gatsby

[#8]

I found out this open ticket - https://issues.cloudera.org/browse/IMPALA-1575

Probably the query you saw is from Hue.

Gatsby