Summary

Are you having issues with more queries being handled by a single Impala Coordinator?

Does this eventually lead to OOM scenarios?

Let’s consider you have 3 Impala Coordinators within your cluster and notice that there are queries that skew onto any one of the Impala Coordinators and overwhelm it.

Note how one of the Impala Coordinators in the above example has 73 running queries, and the other 2 have relatively few.

Investigation

Source IP Persistence

To ascertain why any Impala Coordinator can skew the number of running queries that are active on it, look at the way the proxy is set up to handle incoming queries.

‘Source IP Persistence’ means setting up sessions from the same IP address to always go to the same coordinator. This setting is required when setting up high availability with Hue. It is also required to avoid the Hue message ‘results have expired’, which indicates when a query is sent to the cluster on one coordinator but the result doesn’t return to the user via the same coordinator/Hue Server.

Example HAProxy Configuration for Source IP Persistence

The public docs for setting up HAProxy for Impala - Configuring Load Balancer for Impala. 

Example setup of Hue-Impala connectivity within /etc/haproxy/haproxy.cfg as follows:

Now let’s review what can impact the overall connection count into an Impala Coordinator: Hue, Hive & Impala timeout settings.

Example Timeout Settings

The following settings might mimic what you have currently set within your Hue, Hive & Impala services.

Hue

Hive

Impala

Proposed Timeout Settings

Whilst the actual settings will vary cluster by cluster, we recommend moving away from the default settings and setting all of the idle parameters to 2 hours across the board in all 3 services: Hue, Hive & Impala.

This is an initial goal of introducing timeouts whilst monitoring the user experience.  The ultimate best practice in this area is to head toward having:

• Idle Query Timeouts of 300 seconds (or 5 minutes)
• Idle Session Timeouts of 600 seconds (or 10 minutes)

NOTE - all of the parameters being discussed relate to ‘idle’ sessions and queries; in other words, the user has to have left either the session or query in an idle state before the idle parameters will kick in. No active session or query will be captured by this change in the service(s) behavior (s).

Resolution

Hue

Steps to perform:

• Go to CM - Hue - Configuration
• Search for “Auto Logout Timeout”
• Change to 2 hours
• Restart Hue Service

Hive

Steps to perform:

• Go to CM - Hive - Configuration
• Search for “Idle Operation Timeout”
• Change to 300 seconds
• Search for “Idle Session Timeout”
• Change to 600 seconds
• Restart Hive Service

Hive on Tez

Steps to perform:

• Go to CM - Hive on Tez - Configuration
• Search for “Idle Operation Timeout”
• Change to 300 seconds
• Search for “Idle Session Timeout”
• Change to 600 seconds
• Restart Hive on Tez Service

Impala

Steps to perform:

• Go to CM - Impala - Configuration
• Search for “Idle Query Timeout”
• Change to 300 seconds
• Search for “Idle Session Timeout”
• Change to 600 seconds
• Restart Impala Service