question Re: Hadoop Log Monitoring in Support Questions

Hadoop Log Monitoring

ravi1 — Fri, 16 Sep 2022 09:46:36 GMT

Do we have a list of things that can be monitoring from hadoop logs (datanode/nodemanager/namenode/resourcemanager)? Right now, I am working on ingesting logs into Kibana and they monitor Errors, Exceptions and application statistics. Are there any other things that we can get from these logs or has someone already worked on these logs to gain some intelligence on working of the cluster?

Re: Hadoop Log Monitoring

nsabharwal — Thu, 29 Oct 2015 19:09:21 GMT

@ravi@hortonworks.com

This may not be relevant but sharing in case it rings bell for new ideas link

As you mentioned already, I am going to rephrase with more words/context

Errors - Critical alert, Sev1

Warning - Sev2

Info: Sev4

This is good repo

Re: Hadoop Log Monitoring

ravi1 — Fri, 30 Oct 2015 05:09:14 GMT

We are just not thinking on the lines of errors and warnings from logs. Its more in the lines of things that are in logs that are not in metrics. Maybe its good to somehow put them into metrics at some point. But till that time, we would like to read things like timeouts if there are any, fetcher times etc.

Re: Hadoop Log Monitoring

sunile_manjee — Mon, 13 Mar 2017 08:54:00 GMT

HDP service logs are available in ambari log search. the back end is solr so you can pull all or only relevant info based on your requirements. Also for service level metrics, ambari stores these now in grafana.