question Hit\Miss Writing To Syslog System in Support Questions

Hit\Miss Writing To Syslog System

jame1997 — Fri, 29 Mar 2024 02:08:27 GMT

Hello,
I'm currently running Nifi version 1.8.0, starting from last week I noticed that I'm missing PutSysLog message from my syslog system. Some logs would appear and some wouldn't write to the syslog system. I checked my syslog system and was unable to find any issue. No other systems are experiencing issue writing to the syslog system. I enabled DEBUG on PutSysLog and it shows it was successful in sending the message to the syslog system but no entry exists. At the same time other PutSysLog does record the entries in syslog system and the file flows through other processors. Any suggestions on how to troubleshoot this issue?
Thanks!

Re: Hit\Miss Writing To Syslog System

MattWho — Fri, 29 Mar 2024 14:13:23 GMT

@jame1997

My first question would be how you have your putSyslog processor configured?

Are you using TCP or UDP. If you are using UDP there is not going to be any confirmed delivery. That is not a lossless protocol. While TCP does have confirmed delivery at the expense of speed.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Hit\Miss Writing To Syslog System

jame1997 — Fri, 29 Mar 2024 15:15:32 GMT

Hello Matt,
Its configured via UDP, I understood that there is no way to track data loss. Is there anything I can check from Nifi that may contribute to this issue ? For example if systems resource are high or other factor.

Thanks!

Re: Hit\Miss Writing To Syslog System

MattWho — Fri, 29 Mar 2024 17:23:03 GMT

@jame1997

Not much to look at from a NiFi side. NiFi is writing to the network successfully and there is some loss then happening between NiFi and your syslog server. Resource usage affecting your NiFi would only slow down processing but not result in dataloss within NiFi. So the PutSyslog would successfully write all bytes to the network before passing the FlowFile to the "success" relationship. Using TCP of course would allow NiFi to confirm successful delivery thus allowing NiFi to appropriately retry, or route to either failure or success relationships.

You could look at the data rate NiFi is writing from the putSyslog by looking at the stats on the processor. Then maybe you could experiment with:
1. netstat - nsu to check for UDP packet loss.
2. using a network monitoring tool

Thank you,
Matt