Cloudera Impala session getting hanged intermittently

[#1]

Hi,

Recently in one of our critical Cloudera CDP 7.1.7_p1044 cluster we could see that Impala is getting hanged intermittently and whenever Impala session are getting hanged no one was being able to run even Show databases and all Impala sessions are getting hanged. While restarting the Impala Catalog service the issue gets resolved but again it reappears after 5-6 hours or may be after 1 days as there is no fixed time.

We checked the Impala catalog log and found the following errors when it is in hung state:

"Error getting policies; Received NULL response!!. secure Mode=true, user=impala/[email protected] (auth: KERBEROS), service Name=hive"

Also while checking the Ranger errors we got the following error :

"RangerAdminRESTClient.java:167] Error getting policies; Received NULL response!!"

Also getting error in Hive Server 2 logs:

"Error occurred during processing of message.
SSL Handshake Exception: Remote host terminated the handshake"

Any help will be highly appreciated.

Thanks.

[#2]

Welcome to the community @learndeb. Perhaps @BennyZ @James G or @Chella will have some input on this one. 

[#3]

Hi,

Is Impala session hang means not able to run any queries? Is this issue observed via hue or impala-shell?

repro the issue via impala-shell and check the coordinator logs to get any suspected error messages. 

At the time of issue check the cluster load how may queries running concurrently, Issue may arise if any expensive/complex operations are running.

Do not frequently run "invalidate metadata" query in the cluster it is one of the expensive operations and it pressurises the catalog service.

Regards,

Chethan YM

[#4]

Hi @learndeb  how this case got resolved?

[#5]

Hello  @du_datafactory  ,

Thank you for reaching out to the Cloudera Community.

May I ask if you're experiencing the same issue that @learndeb  reported, or are you seeing a different error?

Based on the logs shared in the original post, the issue appears to be related to the communication between Impala/HiveServer2 and Ranger, rather than an issue with Impala query processing itself. The "Received NULL response" while downloading Ranger policies is typically a secondary symptom that can occur if Impala or HiveServer2 is unable to successfully authenticate (for example, due to TLS or Kerberos authentication issues) before retrieving authorization policies from Ranger Admin.

If you could share the exact error you're encountering, along with any initial troubleshooting you've performed (for example, relevant Impala, HiveServer2, or Ranger logs), we'd be happy to help you investigate the issue further.