ELastic log search & Kibana installation and configuration

[#1]

Looking for a good documentation of how to install ELK, LogStash and Kibana on a Ambari managed hadoop platform.

My versions are:

Centos 7.x

Ambari 2.5.1

HDP 2.4.3

There are some documents on GitHub but not sure if that works. Any success stories here ?

[#2]

@Prakash Punj best way to setup logstash with input as hadoop log files and filter the fields in filter section in logstash configuration itself and export visualize in Kibana !

this may help you with configuration.

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals

[#3]

Check out CyberSecurity HCP at https://docs.hortonworks.com/.

I had pretty good success with that setup. It brings elasticsearch (5.2) and kibana into your ambari cluster with an elasticsearch-mpack. I took that mpack and created my own mpack for Elasticsearch (6.3) , Logstash (6.3) , Kibana (6.3) , Filebeat (6.3) , Metricbeat (6.3)

[#4]

Hi, @Steven Matison can you provide some instruction. How I can repeate this action on HDP or HCP. It will be very helpfully for all of as.

[#5]

HCP stack at HW has full documentation of how to install that elastic mpack. You should start there.

Taking that to current version (6.3.0 and most recently 6.3.2) was quite a bit of work but if you unpack the mpack and look at the actual files, it really isnt that complicated. I was thinking about creating a tutorial/article here about my ELK Stack Mpack, but I do not enough reputation to create those types of articles.

[#6]

Looking for good ELK installation related document also please suggest the best fit version of elastic search for our cluster.

Ambari and HDP versions are as below :-

Ambari 2.7.1

HDP 3.0.1

[#7]

@rajendra look here for some articles I am working on for Mpack for HDP 3:

https://community.hortonworks.com/content/kbentry/242228/create-a-test-base-for-elk-stack-632.html

and the parent article:

https://community.hortonworks.com/articles/215078/how-to-install-elk-stack-632-in-ambari.html