Support Questions

Find answers, ask questions, and share your expertise

queries in flight issue

avatar
Explorer

Hey all,

I'm using Impala 2.2.0, cdh 5.4.3.

 

There are times that I have queries that had been canceled by the user, but they're appear in the deamon web UI in "Queries in flight" tab.

 

 

ClouderaManagerQueriesExecutingStatus.PNGQueriesInFlightDeamonWebUI.PNG

 

These queries in EXCEPTION state, and when I look into their profile, I can see that the status is "Cancelled".

 

QueryProfile.PNG

 

Sounds great, BUT - when I look deeper, and see the memory usage for the specific query, It looks like the cancel command didn't free the memory.

 

QueryMemoryUsage.PNG

 

CombinedRamUsageCluster.PNG

In case of massive queries, this issue harm concurrency, and the coordinator host become a bottleneck.

(As u can see in the attached graph).

 

Any ideas?

 

Thanks

 Dror

 

 

 

7 REPLIES 7

avatar
Rising Star

You might need to click the `cancel` link next to query.

 

Although a query is under exception status, it seems still running and using resources

 

Gatsby

avatar
Explorer

Thanks @thewayofthinkin.
I'm aware for the cancel button, but in production env, where you can have massive amount of queries in EXCEPTION mode, canceling queries manually is not the right solution.

 

I wonder if it can be fixed by configuration or script.

Anyway, it looks like a bug to me.

 

Have anyone experienced this issue?

If yes, I would like to hear what was your action items.

 

Thanks

Dror

 

 

avatar
Super Collaborator

Hi dsss,

 

Can you tell us which application issues the Cancel command (JDBC, ODBC)? Can you also check the log files to see what's going on? Usually grepping the logs for the query id may lead to stuck fragments or similar.

 

Cheers, Lars

avatar
Rising Star

@dsss

 

yeap. I agree with you. we can't just kill those quries manually.

I think most of times quries in exception status and in filght quries from Hue.

 

@Lars Volker

Is there way to log slow quries like MySQL slow quries?

Is there way to set timeout for long running quries so that those quries can be killed?

 

Thank you

Gatsby

avatar
Super Collaborator

@thewayofthinkin

 

This page has information on various timeout settings for Impala: http://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html

 

If you're using CM, then all queries should be logged and you can search for long running queries in CM. Without CM I don't know of any convenient helper, so you may have to search/filter the logfiles yourself.

avatar
Rising Star

Thank you.

 

I will check with team to set the timeout.

 

Gatsby

avatar
Rising Star

I found out this open ticket - https://issues.cloudera.org/browse/IMPALA-1575

Probably the query you saw is from Hue.

 

Gatsby