question Re: Hadoop Ecosystem Log Monitoring and Analysis in Archives of Support Questions (Read Only)

Hadoop Ecosystem Log Monitoring and Analysis

sedatkestepe — Fri, 10 Feb 2017 19:18:16 GMT

Hi,

I am about to setup an Hadoop ecosystem. I have some practices with the environment setup but not with monitoring/managing the logs with an external solution.

Since there will be a lot of applications to print logs and yet so much log files, I will need a solution which will be able to monitor them (Will be great if I can also manage and analyze them). I have heard about ELK and some others but I would like to hear from some experienced people.

Can you recommend one such solution?

Btw I'll use Ambari so it'd be good if you add its compatibility information whether or not can be managed by Ambari.

Re: Hadoop Ecosystem Log Monitoring and Analysis

aervits — Fri, 10 Feb 2017 19:59:20 GMT

In past couple of versions of Ambari, we introduced Log Search UI to help you do just that, you can view across all of the logs in the HDP stack from a single viewer. http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/ch_ambari_log_search.html

It parses the logs, indexes them and you can do searches on log levels

Re: Hadoop Ecosystem Log Monitoring and Analysis

sedatkestepe — Fri, 10 Feb 2017 20:08:41 GMT

Hi @Artem Ervits,

Thank you for your answer.

Is that possible to integrate Log Search UI with Kibana?

Re: Hadoop Ecosystem Log Monitoring and Analysis

aervits — Fri, 10 Feb 2017 22:22:35 GMT

@Sedat Kestepe probably can, we don't restrict you from taking the logs and do your own analysis on it, but why would you want to? Take a look at the rest of the documentation, there is a cool UI to slice and dice the logs. Here's a screenshot https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/viewing_service_logs.html

please consider accepting the answer.