Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

queries in flight issue

queries in flight issue

Explorer

Hey all,

I'm using Impala 2.2.0, cdh 5.4.3.

 

There are times that I have queries that had been canceled by the user, but they're appear in the deamon web UI in "Queries in flight" tab.

 

 

ClouderaManagerQueriesExecutingStatus.PNGQueriesInFlightDeamonWebUI.PNG

 

These queries in EXCEPTION state, and when I look into their profile, I can see that the status is "Cancelled".

 

QueryProfile.PNG

 

Sounds great, BUT - when I look deeper, and see the memory usage for the specific query, It looks like the cancel command didn't free the memory.

 

QueryMemoryUsage.PNG

 

CombinedRamUsageCluster.PNG

In case of massive queries, this issue harm concurrency, and the coordinator host become a bottleneck.

(As u can see in the attached graph).

 

Any ideas?

 

Thanks

 Dror

 

 

 

7 REPLIES 7
Highlighted

Re: queries in flight issue

Contributor

You might need to click the `cancel` link next to query.

 

Although a query is under exception status, it seems still running and using resources

 

Gatsby

Re: queries in flight issue

Explorer

Thanks @thewayofthinkin.
I'm aware for the cancel button, but in production env, where you can have massive amount of queries in EXCEPTION mode, canceling queries manually is not the right solution.

 

I wonder if it can be fixed by configuration or script.

Anyway, it looks like a bug to me.

 

Have anyone experienced this issue?

If yes, I would like to hear what was your action items.

 

Thanks

Dror

 

 

Re: queries in flight issue

Expert Contributor

Hi dsss,

 

Can you tell us which application issues the Cancel command (JDBC, ODBC)? Can you also check the log files to see what's going on? Usually grepping the logs for the query id may lead to stuck fragments or similar.

 

Cheers, Lars

Re: queries in flight issue

Contributor

@dsss

 

yeap. I agree with you. we can't just kill those quries manually.

I think most of times quries in exception status and in filght quries from Hue.

 

@Lars Volker

Is there way to log slow quries like MySQL slow quries?

Is there way to set timeout for long running quries so that those quries can be killed?

 

Thank you

Gatsby

Re: queries in flight issue

Expert Contributor

@thewayofthinkin

 

This page has information on various timeout settings for Impala: http://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html

 

If you're using CM, then all queries should be logged and you can search for long running queries in CM. Without CM I don't know of any convenient helper, so you may have to search/filter the logfiles yourself.

Re: queries in flight issue

Contributor

Thank you.

 

I will check with team to set the timeout.

 

Gatsby

Re: queries in flight issue

Contributor

I found out this open ticket - https://issues.cloudera.org/browse/IMPALA-1575

Probably the query you saw is from Hue.

 

Gatsby

Don't have an account?
Coming from Hortonworks? Activate your account here