Created 09-28-2017 03:52 PM
Hi,
I wanted to know if ListenSyslog processor can handle around 3M messages per second(around 3 Gigs per second)? I tried with around 2K messages per second it works smooth, but I just wanted to understand if such huge data can be handled?
Can someone help me to understand
Created 09-29-2017 01:46 PM
Hi,
Can you help me understand what is the role of syslog-ng in your configuration?
Is syslog-ng being used to feed syslog messages to the ListenSyslog processor?
syslog-ng can also write directly to HDFS (https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-hdfs.html) and here you can find details about its performance and tips for tuning its performance:
https://www.balabit.com/blog/testing-performace-log-streaming-hdfs-syslog-ng/
,Hi,
You refer in the question to syslog-ng, can you clarify what is the role of syslog-ng in your configuration?
Is syslog-ng being used to feed syslog messages to the ListenSyslog processor?
syslog-ng can also directly write messages to HDFS (https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-hdfs.html) and you can find performance measurement results and optimisation tips here:
https://www.balabit.com/blog/testing-performace-log-streaming-hdfs-syslog-ng/
Created 10-13-2017 10:30 AM
Thanks for your help and sorry for turning up late.
I am using ListenSyslog to put messages into hdfs after extraction and to hive tables.
So, to check the max number of messages can be handled by Nifi, have asked this question.
Created 10-13-2017 04:49 PM
It is pretty much impossible to say what a maximum number of messages is because there a ton of variables that can impact performance... disks, processors, size of messages, etc.
For this volume of messages you will likely want a NiFi cluster and some way to load balance the data from your producers so that it is evenly distributed to the ListenSyslog on each node of your cluster.