Support Questions
Find answers, ask questions, and share your expertise

Hadoop Ecosystem Log Monitoring and Analysis

Solved Go to solution

Hadoop Ecosystem Log Monitoring and Analysis

Rising Star

Hi,

I am about to setup an Hadoop ecosystem. I have some practices with the environment setup but not with monitoring/managing the logs with an external solution.

Since there will be a lot of applications to print logs and yet so much log files, I will need a solution which will be able to monitor them (Will be great if I can also manage and analyze them). I have heard about ELK and some others but I would like to hear from some experienced people.

Can you recommend one such solution?

Btw I'll use Ambari so it'd be good if you add its compatibility information whether or not can be managed by Ambari.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Hadoop Ecosystem Log Monitoring and Analysis

Mentor

In past couple of versions of Ambari, we introduced Log Search UI to help you do just that, you can view across all of the logs in the HDP stack from a single viewer. http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/ch_ambari_log_s...

It parses the logs, indexes them and you can do searches on log levels

View solution in original post

3 REPLIES 3

Re: Hadoop Ecosystem Log Monitoring and Analysis

Mentor

In past couple of versions of Ambari, we introduced Log Search UI to help you do just that, you can view across all of the logs in the HDP stack from a single viewer. http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/ch_ambari_log_s...

It parses the logs, indexes them and you can do searches on log levels

View solution in original post

Re: Hadoop Ecosystem Log Monitoring and Analysis

Rising Star

Hi @Artem Ervits,

Thank you for your answer.

Is that possible to integrate Log Search UI with Kibana?

Re: Hadoop Ecosystem Log Monitoring and Analysis

Mentor

@Sedat Kestepe probably can, we don't restrict you from taking the logs and do your own analysis on it, but why would you want to? Take a look at the rest of the documentation, there is a cool UI to slice and dice the logs. Here's a screenshot https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/viewing_servic...

please consider accepting the answer.