Cloudera Impala session getting hanged intermittently

[#1]

Hi,

Recently in one of our critical Cloudera CDP 7.1.7_p1044 cluster we could see that Impala is getting hanged intermittently and whenever Impala session are getting hanged no one was being able to run even Show databases and all Impala sessions are getting hanged. While restarting the Impala Catalog service the issue gets resolved but again it reappears after 5-6 hours or may be after 1 days as there is no fixed time.

We checked the Impala catalog log and found the following errors when it is in hung state:

"Error getting policies; Received NULL response!!. secure Mode=true, user=impala/pc1133.aaa.abc.com@corp.xxx.COM (auth: KERBEROS), service Name=hive"

Also while checking the Ranger errors we got the following error :

"RangerAdminRESTClient.java:167] Error getting policies; Received NULL response!!"

Also getting error in Hive Server 2 logs:

"Error occurred during processing of message.
SSL Handshake Exception: Remote host terminated the handshake"

Any help will be highly appreciated.

Thanks.

[#2]

Welcome to the community @learndeb. Perhaps @BennyZ @James G or @Chella will have some input on this one. 

[#3]

Hi,

Is Impala session hang means not able to run any queries? Is this issue observed via hue or impala-shell?

repro the issue via impala-shell and check the coordinator logs to get any suspected error messages. 

At the time of issue check the cluster load how may queries running concurrently, Issue may arise if any expensive/complex operations are running.

Do not frequently run "invalidate metadata" query in the cluster it is one of the expensive operations and it pressurises the catalog service.

Regards,

Chethan YM