We are brainstorming in my team to choose de correct solution for log collecting. We're used to Filebeat / Elasticsearch / Logstash / Kibana but we are using NiFi a lot for many use cases. We're wondering if it would be viable to replace logstash with NiFi, but we can't find any usage feedback for Filebeat / Nifi / Elasticsearch / Kibana.
So I have several questions :
- Have any of you use this solution ? Do you recommend it ? Have you faced any issue ?
- Does the ListenBeats works well, do you guys have any feedback using it on production ?
- Do you recommend using Minifi instead of Filebeat ? We may have to send our logs directly to Kafka in a near future, do you think Minifi would, in the end, be the most fit for the job ?
Sorry for the long post, and thank you for your answers.
MiNiFi Agent either C++ or Java has many advantages over Filebeat including letting you program easily in a GUI using Edge Flow Manager. It also does more than grab logs, it can do analysis, conversion, filtering and more advanced processing. It can also read SYSLOG, databases, MQTT, JMS and a hundred other things.
We don't use any ELK stuff, NiFi replaces all of that. NiFi is in production at hundreds of massive companies for enterprise critical applications.
NiFi can push to ElasticSearch. You can store all of your data to HDFS and use SOLR on top for searches.