Apache NiFi ListenSyslog drop incoming data when InvokeHTTP tries to resend failure data

Question

Hello Community,I have a configuration where ListenSysylog is listenig and provides the received data to InvokeHTTP. Also I use a funnel to redirect Http failures to the InvokeHttp once again. The problem occurs when InvokeHttp tries to resend data and it consumes so many threads that ListenSyslog starts droping incoming data. I do not want to lost any incoming data and in the same time I want to be sure that data will be send. Do you have a similar problem? Is there any way to limit proccessors?Thanks

araujo · Accepted Answer

@Curry5103 ,

ListenSyslog can receive a very large amount of data and it may be hard to match the throughput of that with an InvokeHTTP processor.

To avoid dropping syslog messages, especially if using UDP protocol, you will probably need to provide enough buffer for those messages to decouple the receipt of the messages from the InvokeHTTP execution. You can do that in NiFi by increase the maximum queue size limits, or maybe use something like Kafka, where you can temporarily store the syslog messages and another flow can read from Kafka and call InvokeHTTP for each of them.

Cheers,

André

ckumar · Answer

This line "The problem occurs when InvokeHttp tries to resend data and it consumes so many threads that ListenSyslog starts dropping incoming data"

My understanding is that InvokeHttp will try run with max number of threads based upon what is configured concurrent task settings, beyond that it will not use any more threads at the same ListenSyslog is also running with it own concurrent task setting and max threads it can use and which can be further tune to hold more request based on settings Max Size of Message Queue and Max Size of Socket Buffer and Max Number of TCP Connections.

To hold your initial anaylsys between InvokeHTTP and ListenSyslog you can provide more details around your flow design when the issue occurs like screen shots of flow, configs setting for ListenSyslg and InvokeHTTP would help more to understand.

Thank You

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.