Created on 01-05-2018 06:02 PM - edited 09-16-2022 05:42 AM
Looking for a good documentation of how to install ELK, LogStash and Kibana on a Ambari managed hadoop platform.
My versions are:
There are some documents on GitHub but not sure if that works. Any success stories here ?
Created 07-18-2018 05:25 AM
@Prakash Punj best way to setup logstash with input as hadoop log files and filter the fields in filter section in logstash configuration itself and export visualize in Kibana !
this may help you with configuration.
Created on 07-20-2018 06:19 PM - edited 08-18-2019 02:06 AM
Check out CyberSecurity HCP at https://docs.hortonworks.com/.
I had pretty good success with that setup. It brings elasticsearch (5.2) and kibana into your ambari cluster with an elasticsearch-mpack. I took that mpack and created my own mpack for Elasticsearch (6.3) , Logstash (6.3) , Kibana (6.3) , Filebeat (6.3) , Metricbeat (6.3)
Created 08-23-2018 02:32 PM
Hi, @Steven Matison can you provide some instruction. How I can repeate this action on HDP or HCP. It will be very helpfully for all of as.
Created 08-23-2018 06:44 PM
HCP stack at HW has full documentation of how to install that elastic mpack. You should start there.
Taking that to current version (6.3.0 and most recently 6.3.2) was quite a bit of work but if you unpack the mpack and look at the actual files, it really isnt that complicated. I was thinking about creating a tutorial/article here about my ELK Stack Mpack, but I do not enough reputation to create those types of articles.
Created 02-27-2019 09:24 AM
Looking for good ELK installation related document also please suggest the best fit version of elastic search for our cluster.
Ambari and HDP versions are as below :-
Created 03-04-2019 08:39 PM
@rajendra look here for some articles I am working on for Mpack for HDP 3:
and the parent article: