Hadoop Ecosystem Log Monitoring and Analysis

[#1]

Hi,

I am about to setup an Hadoop ecosystem. I have some practices with the environment setup but not with monitoring/managing the logs with an external solution.

Since there will be a lot of applications to print logs and yet so much log files, I will need a solution which will be able to monitor them (Will be great if I can also manage and analyze them). I have heard about ELK and some others but I would like to hear from some experienced people.

Can you recommend one such solution?

Btw I'll use Ambari so it'd be good if you add its compatibility information whether or not can be managed by Ambari.

[#2 · Accepted solution]

In past couple of versions of Ambari, we introduced Log Search UI to help you do just that, you can view across all of the logs in the HDP stack from a single viewer. http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/ch_ambari_log_s...

It parses the logs, indexes them and you can do searches on log levels

[#3]

Hi @Artem Ervits,

Thank you for your answer.

Is that possible to integrate Log Search UI with Kibana?

[#4]

@Sedat Kestepe probably can, we don't restrict you from taking the logs and do your own analysis on it, but why would you want to? Take a look at the rest of the documentation, there is a cool UI to slice and dice the logs. Here's a screenshot https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/viewing_servic...

please consider accepting the answer.