Receiving Error "EAGAIN (timed out)"

[#1]

Every so often we get the following error from Impala:

[HY000] [Cloudera][ThriftExtension] (11) Error occurred while contacting server: EAGAIN (timed out). The connection has been configured to use a binary mechanism for authentication. This error might occur if the server is not using a binary mechanism for authentication. (11) (SQLDriverConnect)

Our software makes up to 3 attempts to run the action, if the first (and second) attempts fail.

The attempt fails on this error sometimes, but not always.

This (the fact that the error occurs only some of the times) leads us to think that it is not a configuration issue, as suggested by the wording of the error.

1. What causes this error ?

2. How should we deal with this ?
3. Can we prevent the problem from happening ?

Thanks

  Lior H.

[#2]

@liorh 

The error message you're encountering, "[HY000] [Cloudera][ThriftExtension] (11) Error occurred while contacting server: EAGAIN (timed out)," typically indicates a timeout issue while attempting to establish a connection with the Impala server using the Thrift protocol.

Below is just generic troubleshooting tips, You need to analyse and troubleshoot your environment.

1. Causes of the Error:

   • Network Latency: The error may occur due to network latency or connectivity issues between your application and the Impala server. This can lead to timeouts during the connection attempt.
   • Server Load: If the Impala server is under heavy load or experiencing resource constraints, it may not be able to handle incoming connection requests promptly, resulting in timeouts.
   • Thrift Protocol Issues: The error message mentions using a binary mechanism for authentication. If there are inconsistencies or misconfigurations in the Thrift protocol settings between your application and the Impala server, it could lead to connection failures.

2. Dealing with the Error:

   • Retry Mechanism: As you mentioned, implementing a retry mechanism in your software to make multiple attempts to run the action is a good approach. This can help mitigate transient network issues or server load spikes that may cause the initial connection attempt to fail.

3. Preventing the Problem:

   • Optimize Network Configuration: Review and optimize the network configuration between your application and the Impala server to minimize latency and improve reliability. This may include configuring network settings, optimizing routing, or using dedicated network connections.
   • Server Performance Tuning: Monitor the performance of the Impala server and address any resource bottlenecks or performance issues that could lead to connection timeouts. This may involve optimizing server configuration, increasing resources, or tuning Impala parameters.
   • Thrift Protocol Configuration: Ensure that the Thrift protocol settings, including authentication mechanisms, are correctly configured and consistent between your application and the Impala server. 

Review the trace level driver logs, Impalad, statestore and catalog logs at the time of issue to see if we can get some details of the issue.

Regards,

Chethan YM