Changing Impala-shell session timeout

[#1]

I'm executing 300+ queries at once via impala-shell in a pool that allows a queue of 500. However, after 10 minutes I start to see some queries flagged as error with a query status of "session closed". The query uses the following option "SET IDLE_SESSION_TIMEOUT=3600;" Any thoughts on what property I need to change so the session stays open longer than 10 minutes?

I'm running Impala Shell v2.12.0-cdh5.15.2

[#2]

Sessions with queries that are queued in admission control generally shouldn't be considered be considered idle. What Impala server version are you connected to?

What does your script look like? Are you running 300+ instances of impala-shell at the same time?

[#3]

I'm connected to... 

impalad version 2.12.0-cdh5.15.2

And yes, I'm running 300+ instances of impala-shell at the same time. I have a python script that loops through a set of queries and executes one impala-shell command per query:

impala-shell -k -i edpprd --ssl --quiet -B -c -q "SET REQUEST_POOL=root.pool1; SET IDLE_SESSION_TIMEOUT=3600; select 'table1' as table_name, count(1) as cloud_count from default.table"

Some of the queries take 30+ minutes to complete when running by themselves. I also tried this without defining a pool and just submitting all the queries on their own and it overwhelms the impala server memory and results in 100+ failed queries. 

[#4]

Hi @EricC

Are you able to share some of the failed impala query PROFILE for review? It might tell some story from there.

Cheers
EricL

[#5]

I ended up submitting this as a Cloudera Support request. I will post the solution once we figure it out.

[#6]

So IDLE_SESSION_TIMEOUT doesn't actually do anything if you set it in impala-shell. There's a technical distinction here - "set" in impala-shell is implemented in the shell itself, then send along with the queries. Whereas if you run "set" as a statement through, say, the JDBC driver, then "set" is run as a server-side command that modifies the session state.

I tried to reproduce the issue on my local Impala environment, but couldn't. I was watching the /sessions page in the debug page and I could see the "last accessed" time get continually incremented while the query was in the queue. I think any timeout issue probably got indirectly fixed by https://issues.apache.org/jira/browse/IMPALA-5216 in CDH6.1/Impala 3.1, since the query state would be polled directly by impala-shell while the query is in the queue, rather than having impala-shell blocked waiting for a response.

Thanks for submitting the support request, that should help get to the bottom of this.