- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Using NiFi instead of Logstash
- Labels:
-
Apache MiNiFi
-
Apache NiFi
Created ‎04-25-2019 12:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi guys,
We are brainstorming in my team to choose de correct solution for log collecting. We're used to Filebeat / Elasticsearch / Logstash / Kibana but we are using NiFi a lot for many use cases. We're wondering if it would be viable to replace logstash with NiFi, but we can't find any usage feedback for Filebeat / Nifi / Elasticsearch / Kibana.
So I have several questions :
- Have any of you use this solution ? Do you recommend it ? Have you faced any issue ?
- Does the ListenBeats works well, do you guys have any feedback using it on production ?
- Do you recommend using Minifi instead of Filebeat ? We may have to send our logs directly to Kafka in a near future, do you think Minifi would, in the end, be the most fit for the job ?
Sorry for the long post, and thank you for your answers.
Created ‎04-25-2019 12:44 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
MiNiFi Agent either C++ or Java has many advantages over Filebeat including letting you program easily in a GUI using Edge Flow Manager. It also does more than grab logs, it can do analysis, conversion, filtering and more advanced processing. It can also read SYSLOG, databases, MQTT, JMS and a hundred other things.
We don't use any ELK stuff, NiFi replaces all of that. NiFi is in production at hundreds of massive companies for enterprise critical applications.
NiFi can push to ElasticSearch. You can store all of your data to HDFS and use SOLR on top for searches.
Created ‎04-25-2019 12:47 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎04-26-2019 07:24 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello, thank you for your answer. There's a lot of useful info in the links you provided.
If anyone has any more feedback to share, It would be very helpful.
Created ‎03-13-2023 08:42 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am trying to use Nifi to send data via tcp to relevant filebeat modules, but I am finding that Nifi is adding unwanted data to the logs. Has anyone come across this problem if using Nifi to send to Elasticsearch before and if so, what might be a good solution to preventing Nifi from doing this? (separate pipeline in Elastic to drop/rename/add etc or to use Logstash from Nifi or add a new module to not pick up Nifi data...?)
