Skip to main content
jame1997
jame1997
Silver ⭐️⭐️⭐️
March 29, 2024
Solved

Hit\Miss Writing To Syslog System

Hello,
I'm currently running Nifi version 1.8.0, starting from last week I noticed that I'm missing PutSysLog message from my syslog system.  Some logs would appear and some wouldn't write to the syslog system.  I checked my syslog system and was unable to find any issue.  No other systems are experiencing issue writing to the syslog system.  I enabled DEBUG on PutSysLog and it shows it was successful in sending the message to the syslog system but no entry exists. At the same time other PutSysLog does record the entries in syslog system and the file flows through other processors.      Any suggestions on how to troubleshoot this issue?
Thanks!

    MattWho
    MattWhoAnswer
    Titanium ⭐️⭐️⭐️
    March 29, 2024

    @jame1997 

    Not much to look at from a NiFi side. NiFi is writing to the network successfully and there is some loss then happening between NiFi and your syslog server.   Resource usage affecting your NiFi would only slow down processing but not result in dataloss within NiFi.  So the PutSyslog would successfully write all bytes to the network before passing the FlowFile to the "success" relationship.  Using TCP of course would allow NiFi to confirm successful delivery thus allowing NiFi to appropriately retry, or route to either failure or success relationships.

    You could look at the data rate NiFi is writing from the putSyslog by looking at the stats on the processor.  Then maybe you could experiment with:
    1. netstat - nsu to check for UDP packet loss.  
    2. using a network monitoring tool


    Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

    Thank you,
    Matt

     

    View in thread

    3 replies

    MattWho
    Titanium ⭐️⭐️⭐️
    MattWho
    Titanium ⭐️⭐️⭐️
    March 29, 2024

    @jame1997 

    My first question would be how you have your putSyslog processor configured?

    Are you using TCP or UDP.  If you are using UDP there is not going to be any confirmed delivery.  That is not a lossless protocol.  While TCP does have confirmed delivery at the expense of speed.  

    Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

    Thank you,
    Matt



    Translate
      jame1997
      jame1997Author
      Silver ⭐️⭐️⭐️
      March 29, 2024

      Hello Matt,
      Its configured via UDP, I understood that there is no way to track data loss. Is there anything I can check from Nifi that may contribute to this issue ?  For example if systems resource are high or other factor. 

      Thanks!

      Translate
        MattWho
        Titanium ⭐️⭐️⭐️
        MattWhoAnswer
        Titanium ⭐️⭐️⭐️
        March 29, 2024

        @jame1997 

        Not much to look at from a NiFi side. NiFi is writing to the network successfully and there is some loss then happening between NiFi and your syslog server.   Resource usage affecting your NiFi would only slow down processing but not result in dataloss within NiFi.  So the PutSyslog would successfully write all bytes to the network before passing the FlowFile to the "success" relationship.  Using TCP of course would allow NiFi to confirm successful delivery thus allowing NiFi to appropriately retry, or route to either failure or success relationships.

        You could look at the data rate NiFi is writing from the putSyslog by looking at the stats on the processor.  Then maybe you could experiment with:
        1. netstat - nsu to check for UDP packet loss.  
        2. using a network monitoring tool


        Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

        Thank you,
        Matt

         

        Translate
          Terms & ConditionsAccessibility statement