Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ELastic log search & Kibana installation and configuration

ELastic log search & Kibana installation and configuration

Rising Star

Looking for a good documentation of how to install ELK, LogStash and Kibana on a Ambari managed hadoop platform.

My versions are:

Centos 7.x

Ambari 2.5.1

HDP 2.4.3

There are some documents on GitHub but not sure if that works. Any success stories here ?

6 REPLIES 6

Re: ELastic log search & Kibana installation and configuration

New Contributor

@Prakash Punj best way to setup logstash with input as hadoop log files and filter the fields in filter section in logstash configuration itself and export visualize in Kibana !

this may help you with configuration.

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals

Re: ELastic log search & Kibana installation and configuration

Expert Contributor

Check out CyberSecurity HCP at https://docs.hortonworks.com/.

I had pretty good success with that setup. It brings elasticsearch (5.2) and kibana into your ambari cluster with an elasticsearch-mpack. I took that mpack and created my own mpack for Elasticsearch (6.3) , Logstash (6.3) , Kibana (6.3) , Filebeat (6.3) , Metricbeat (6.3)

79536-ambari-elk.jpg

Re: ELastic log search & Kibana installation and configuration

New Contributor

Hi, @Steven Matison can you provide some instruction. How I can repeate this action on HDP or HCP. It will be very helpfully for all of as.

Re: ELastic log search & Kibana installation and configuration

Expert Contributor

HCP stack at HW has full documentation of how to install that elastic mpack. You should start there.

Taking that to current version (6.3.0 and most recently 6.3.2) was quite a bit of work but if you unpack the mpack and look at the actual files, it really isnt that complicated. I was thinking about creating a tutorial/article here about my ELK Stack Mpack, but I do not enough reputation to create those types of articles.

Re: ELastic log search & Kibana installation and configuration

New Contributor

Looking for good ELK installation related document also please suggest the best fit version of elastic search for our cluster.

Ambari and HDP versions are as below :-

Ambari 2.7.1

HDP 3.0.1

Re: ELastic log search & Kibana installation and configuration

Expert Contributor
Don't have an account?
Coming from Hortonworks? Activate your account here