Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

what is the maximum data frequency syslog-ng can handle (like number of messages per second)?

Expert Contributor

Hi,

I wanted to know if ListenSyslog processor can handle around 3M messages per second(around 3 Gigs per second)? I tried with around 2K messages per second it works smooth, but I just wanted to understand if such huge data can be handled?

Can someone help me to understand

3 REPLIES 3

New Contributor

Hi,

Can you help me understand what is the role of syslog-ng in your configuration?

Is syslog-ng being used to feed syslog messages to the ListenSyslog processor?

syslog-ng can also write directly to HDFS (https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-hdfs.html) and here you can find details about its performance and tips for tuning its performance:

https://www.balabit.com/blog/testing-performace-log-streaming-hdfs-syslog-ng/

,

Hi,

You refer in the question to syslog-ng, can you clarify what is the role of syslog-ng in your configuration?

Is syslog-ng being used to feed syslog messages to the ListenSyslog processor?

syslog-ng can also directly write messages to HDFS (https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-hdfs.html) and you can find performance measurement results and optimisation tips here:

https://www.balabit.com/blog/testing-performace-log-streaming-hdfs-syslog-ng/

Expert Contributor

@Istvan Szabo

Thanks for your help and sorry for turning up late.

I am using ListenSyslog to put messages into hdfs after extraction and to hive tables.

So, to check the max number of messages can be handled by Nifi, have asked this question.

It is pretty much impossible to say what a maximum number of messages is because there a ton of variables that can impact performance... disks, processors, size of messages, etc.

For this volume of messages you will likely want a NiFi cluster and some way to load balance the data from your producers so that it is evenly distributed to the ListenSyslog on each node of your cluster.