Hello Harsh, Thank you for the help on this. I was able to identify some information that helped here. Will come back in case need further help. Will accept your reply as Solution. 🙂 Thanks snm1523 ... View more

Thank you for the reply Harsh J. Would you be able to please help me with any quick command / script to identify avoidable open files or files stuck in some process using 'lsof' and guide further actions to take? I tried running a generic 'lsof | grep java' but it obviously gave me a huge list of files and became a bit difficult to get relevant information. Thanks snm1523 ... View more

Hello gzigldrum,   Thank you for the guidance. This certainly helps.   I will go through both the KB articles you shared and also review the Navigator Server and CM Agent logs for audit health warnings.   Will comeback with the findings.   In the meantime, any luck with HIVEMETASTORE_CANARY_HEALTH and NODE_MANAGER_WEB_METRIC_COLLECTION?   Thanks snm1523 ... View more

Hello Gzigldrum,   Thank you for the reply.   Below are the exact health messages from CM for each alert:   HIVEMETASTORE_CANARY_HEALTH: The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a database. OR The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a partition in the table it created. OR The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to drop the table it created.   REGION_SERVER_AUDIT_HEALTH: The health test result for REGION_SERVER_AUDIT_HEALTH has become bad: There is a problem processing audits for REGIONSERVER.   IMPALAD_QUERY_MONITORING_STATUS: he health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 1 error(s) seen monitoring executing queries, and 0 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any.   HIVESERVER2_SCM_HEALTH: The health test result for HIVESERVER2_SCM_HEALTH has become bad: This role s process is starting. This role is supposed to be started.   NAME_NODE_AUDIT_HEALTH: The health test result for NAME_NODE_AUDIT_HEALTH has become bad: There is a problem processing audits for NAMENODE.   NODE_MANAGER_WEB_METRIC_COLLECTION: The health test result for NODE_MANAGER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent is not able to communicate with this role s web server.   These health alerts occur in every 2-3 days in a time frame of 2-3 hours creating around 5-10 tickets during each interval. We have also checked from network side to verify if there was a network outage or a glitch in those windows, however, no luck. We have tried to diagnose through logs of each alert, but haven't found anything interesting.   Hence, looking for some more guidance on what additional could be checked to identify a root cause of these.   Thanks snm1523 ... View more

Thank you for the quick replies, Gzigldrum.   I have created a separate post explaining the alerts we get. Below is the link to the same.   Flooded with failed health test alerts   For every alert we get an incident in BMC as HP OVO is configured to generate an incident and assign it to us.   Every morning we start with close to around 15-20 such incidents followed by 15-20 in rest of the day.   Would be great if you could please post some suggestions to troubleshoot those alerts and help in permanently fixing them.   Note: I will accept your previous reply as solution to this post.   Thanks snm1523 ... View more

Hello Gzigldrum, Thank you for the reply. However, the instructions provided in the KB article and values suggested by you are already applied. Please advise. Thanks Snm1523 ... View more