Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hadoop Log Monitoring

avatar
Guru

Do we have a list of things that can be monitoring from hadoop logs (datanode/nodemanager/namenode/resourcemanager)? Right now, I am working on ingesting logs into Kibana and they monitor Errors, Exceptions and application statistics. Are there any other things that we can get from these logs or has someone already worked on these logs to gain some intelligence on working of the cluster?

1 ACCEPTED SOLUTION

avatar
Master Guru

HDP service logs are available in ambari log search. the back end is solr so you can pull all or only relevant info based on your requirements. Also for service level metrics, ambari stores these now in grafana.

View solution in original post

3 REPLIES 3

avatar
Master Mentor

@ravi@hortonworks.com

This may not be relevant but sharing in case it rings bell for new ideas link

As you mentioned already, I am going to rephrase with more words/context

Errors - Critical alert, Sev1

Warning - Sev2

Info: Sev4

This is good repo

avatar
Guru

We are just not thinking on the lines of errors and warnings from logs. Its more in the lines of things that are in logs that are not in metrics. Maybe its good to somehow put them into metrics at some point. But till that time, we would like to read things like timeouts if there are any, fetcher times etc.

avatar
Master Guru

HDP service logs are available in ambari log search. the back end is solr so you can pull all or only relevant info based on your requirements. Also for service level metrics, ambari stores these now in grafana.