what is the maximum data frequency syslog-ng can handle (like number of messages per second)?

[#1]

Hi,

I wanted to know if ListenSyslog processor can handle around 3M messages per second(around 3 Gigs per second)? I tried with around 2K messages per second it works smooth, but I just wanted to understand if such huge data can be handled?

Can someone help me to understand

[#2]

Hi,

Can you help me understand what is the role of syslog-ng in your configuration?

Is syslog-ng being used to feed syslog messages to the ListenSyslog processor?

syslog-ng can also write directly to HDFS (https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-hdfs.html) and here you can find details about its performance and tips for tuning its performance:

https://www.balabit.com/blog/testing-performace-log-streaming-hdfs-syslog-ng/

,

Hi,

You refer in the question to syslog-ng, can you clarify what is the role of syslog-ng in your configuration?

Is syslog-ng being used to feed syslog messages to the ListenSyslog processor?

syslog-ng can also directly write messages to HDFS (https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-hdfs.html) and you can find performance measurement results and optimisation tips here:

https://www.balabit.com/blog/testing-performace-log-streaming-hdfs-syslog-ng/

[#3]

@Istvan Szabo

Thanks for your help and sorry for turning up late.

I am using ListenSyslog to put messages into hdfs after extraction and to hive tables.

So, to check the max number of messages can be handled by Nifi, have asked this question.

[#4]

It is pretty much impossible to say what a maximum number of messages is because there a ton of variables that can impact performance... disks, processors, size of messages, etc.

For this volume of messages you will likely want a NiFi cluster and some way to load balance the data from your producers so that it is evenly distributed to the ListenSyslog on each node of your cluster.