Using NiFi instead of Logstash

[#1]

Hi guys,

We are brainstorming in my team to choose de correct solution for log collecting. We're used to Filebeat / Elasticsearch / Logstash / Kibana but we are using NiFi a lot for many use cases. We're wondering if it would be viable to replace logstash with NiFi, but we can't find any usage feedback for Filebeat / Nifi / Elasticsearch / Kibana.

So I have several questions :

- Have any of you use this solution ? Do you recommend it ? Have you faced any issue ?

- Does the ListenBeats works well, do you guys have any feedback using it on production ?

- Do you recommend using Minifi instead of Filebeat ? We may have to send our logs directly to Kafka in a near future, do you think Minifi would, in the end, be the most fit for the job ?

Sorry for the long post, and thank you for your answers.

[#2]

MiNiFi Agent either C++ or Java has many advantages over Filebeat including letting you program easily in a GUI using Edge Flow Manager. It also does more than grab logs, it can do analysis, conversion, filtering and more advanced processing. It can also read SYSLOG, databases, MQTT, JMS and a hundred other things.

We don't use any ELK stuff, NiFi replaces all of that. NiFi is in production at hundreds of massive companies for enterprise critical applications.

NiFi can push to ElasticSearch. You can store all of your data to HDFS and use SOLR on top for searches.

[#3]

https://bryanbende.com/development/2015/05/17/collecting-logs-with-apache-nifi

http://www.idata.co.il/2017/11/ingesting-log-data-with-nifi-minifi-and-hive/

https://community.hortonworks.com/articles/218844/log-forwarding-ingestion-patterns-using-minifi-and...

https://community.hortonworks.com/articles/67756/ingesting-log-data-using-minifi-nifi.html

https://www.keedio.org/real-time-data-ingestion-processing-and-storage-of-syslog-using-apache-nifi-f...

[#4]

Hello, thank you for your answer. There's a lot of useful info in the links you provided.

If anyone has any more feedback to share, It would be very helpful.

[#5]

I am trying to use Nifi to send data via tcp to relevant filebeat modules, but I am finding that Nifi is adding unwanted data to the logs. Has anyone come across this problem if using Nifi to send to Elasticsearch before and if so, what might be a good solution to preventing Nifi from doing this? (separate pipeline in Elastic to drop/rename/add etc or to use Logstash from Nifi or add a new module to not pick up Nifi data...?)