Hive Metastore Server Health Bad

[#1]

Hello

Need your help regarding the hive metastore.

we are using cdh 5.14 cluster with hive and other services, we are facing issue with hive metastore.

Every week our hive metastore goes in bad health with alert shows "canary status bad".

when we checked logs of HMS there is nothing suspected.

we have given 78G of mem to the host

when we check the host memory on HMS host  it shows near about 23G physical memory used but it shows the chaches memory used 53 G

Any suggestion on this issue.

- If we do HMS HA, Is it possible both the HMS work parallelly if yes how we can achieve this?

Thanks

Shahezad

[#2]

@Shahezad We do not have any way to put HMS behind a Load Balancers. If we have more than one HMS instances, they only work in  Active-Active mode.

Do consider increasing the metastore client timeout for Service Montitor. 

Also, see if there is any slowness with RDBMS used as metastore DB.

The memory cached in Host level should not be a problem, They get freed when there is a requirement. Ref https://www.baeldung.com/linux/empty-buffer-cache

[#3]

Thanks for replay @smruti 
I will check with your suggestion.