Support Questions
Find answers, ask questions, and share your expertise

PutSQL - Data loss or Retry in a loop

New Contributor

Hello All

I have nifi to extract the attributes of a file then update the results to the database (MariaDB), and it works fine when I try each single file one by one.

The problem is when it is processing bulk of files (20 or more), it intermittently logging somewhere between 50% and 90% of the records only to the DB(very occasionally 100%).

ExtractAttributes - ToJSON - JSONpath - JSONtoSQL - PutSQL

If I look at DB server logs, I can see it is not getting all the INSERT statements from nifi but heaps of SQL COMMIT repeats.

The only error message I see on the app.log is 'Unknown escape sequence' from Java SQL exception (I get this message regardless).

If I place Retry queue back to PutSQL, then I see some better result but still not 100%. Without expire time limits (like 30 secs) it will be on infinite loop.

I will add some screenshots tomorrow when I get back to work.

Thank you