question Re: queries in flight issue in Support Questions

queries in flight issue

dsss — Fri, 16 Sep 2022 10:57:23 GMT

Hey all,

I'm using Impala 2.2.0, cdh 5.4.3.

There are times that I have queries that had been canceled by the user, but they're appear in the deamon web UI in "Queries in flight" tab.

These queries in EXCEPTION state, and when I look into their profile, I can see that the status is "Cancelled".

Sounds great, BUT - when I look deeper, and see the memory usage for the specific query, It looks like the cancel command didn't free the memory.

In case of massive queries, this issue harm concurrency, and the coordinator host become a bottleneck.

(As u can see in the attached graph).

Any ideas?

Thanks

Dror

Re: queries in flight issue

thewayofthinkin — Tue, 31 Jan 2017 23:52:20 GMT

You might need to click the `cancel` link next to query.

Although a query is under exception status, it seems still running and using resources

Gatsby

Re: queries in flight issue

dsss — Wed, 01 Feb 2017 13:27:49 GMT

Thanks @thewayofthinkin.
I'm aware for the cancel button, but in production env, where you can have massive amount of queries in EXCEPTION mode, canceling queries manually is not the right solution.

I wonder if it can be fixed by configuration or script.

Anyway, it looks like a bug to me.

Have anyone experienced this issue?

If yes, I would like to hear what was your action items.

Thanks

Dror

Re: queries in flight issue

Lars Volker — Wed, 01 Feb 2017 15:46:45 GMT

Hi dsss,

Can you tell us which application issues the Cancel command (JDBC, ODBC)? Can you also check the log files to see what's going on? Usually grepping the logs for the query id may lead to stuck fragments or similar.

Cheers, Lars

Re: queries in flight issue

thewayofthinkin — Wed, 01 Feb 2017 17:20:52 GMT

@dsss

yeap. I agree with you. we can't just kill those quries manually.

I think most of times quries in exception status and in filght quries from Hue.

@Lars Volker

Is there way to log slow quries like MySQL slow quries?

Is there way to set timeout for long running quries so that those quries can be killed?

Thank you

Gatsby

Re: queries in flight issue

thewayofthinkin — Thu, 02 Feb 2017 04:17:10 GMT

I found out this open ticket - https://issues.cloudera.org/browse/IMPALA-1575

Probably the query you saw is from Hue.

Gatsby

Re: queries in flight issue

Lars Volker — Thu, 02 Feb 2017 13:35:16 GMT

@thewayofthinkin

This page has information on various timeout settings for Impala: http://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html

If you're using CM, then all queries should be logged and you can search for long running queries in CM. Without CM I don't know of any convenient helper, so you may have to search/filter the logfiles yourself.

Re: queries in flight issue

thewayofthinkin — Thu, 02 Feb 2017 17:28:59 GMT

Thank you.

I will check with team to set the timeout.

Gatsby