Looking for a good documentation of how to install ELK, LogStash and Kibana on a Ambari managed hadoop platform.
My versions are:
There are some documents on GitHub but not sure if that works. Any success stories here ?
@Prakash Punj best way to setup logstash with input as hadoop log files and filter the fields in filter section in logstash configuration itself and export visualize in Kibana !
this may help you with configuration.
Check out CyberSecurity HCP at https://docs.hortonworks.com/.
I had pretty good success with that setup. It brings elasticsearch (5.2) and kibana into your ambari cluster with an elasticsearch-mpack. I took that mpack and created my own mpack for Elasticsearch (6.3) , Logstash (6.3) , Kibana (6.3) , Filebeat (6.3) , Metricbeat (6.3)
Hi, @Steven Matison can you provide some instruction. How I can repeate this action on HDP or HCP. It will be very helpfully for all of as.
HCP stack at HW has full documentation of how to install that elastic mpack. You should start there.
Taking that to current version (6.3.0 and most recently 6.3.2) was quite a bit of work but if you unpack the mpack and look at the actual files, it really isnt that complicated. I was thinking about creating a tutorial/article here about my ELK Stack Mpack, but I do not enough reputation to create those types of articles.
Looking for good ELK installation related document also please suggest the best fit version of elastic search for our cluster.
Ambari and HDP versions are as below :-
@rajendra look here for some articles I am working on for Mpack for HDP 3:
and the parent article: