Hadoop Log Monitoring

[#1]

Do we have a list of things that can be monitoring from hadoop logs (datanode/nodemanager/namenode/resourcemanager)? Right now, I am working on ingesting logs into Kibana and they monitor Errors, Exceptions and application statistics. Are there any other things that we can get from these logs or has someone already worked on these logs to gain some intelligence on working of the cluster?

[#2]

@ravi@hortonworks.com

This may not be relevant but sharing in case it rings bell for new ideas link

As you mentioned already, I am going to rephrase with more words/context

Errors - Critical alert, Sev1

Warning - Sev2

Info: Sev4

This is good repo

[#3]

We are just not thinking on the lines of errors and warnings from logs. Its more in the lines of things that are in logs that are not in metrics. Maybe its good to somehow put them into metrics at some point. But till that time, we would like to read things like timeouts if there are any, fetcher times etc.

[#4 · Accepted solution]

HDP service logs are available in ambari log search. the back end is solr so you can pull all or only relevant info based on your requirements. Also for service level metrics, ambari stores these now in grafana.