My flow is getting data from one database and inputting the data into another. Because I want to Insert/Update the data in the target table I insert the data into a staging table. Afterwards I merge the staging table with the "final" table and truncate the staging table. However if some data gets put into the staging table in the meantime (between merge and truncate) then I lose that data. How could i model the flow to make sure this does not happen? Below is my current "wrong" implementation. ... View more

I have the following Nifi Flow: QueryDatabaseTable -> AvroToJSON -> SplitJSON -> ExtractText (write value to attribute) -> FetchDistributedMapCache (check if the attribute is available in cache, which is set in another flow) -> UpdateRecord (Rename some fields) -> MergeContent -> PutDatabaseRecord And sometimes this error occurs: The FlowFile is not routed to failure, but kept in the queue and the Processor is now stuck in an endless loop trying to process the faulty FlowFile. When i skip the MergeContent/MergeRecord Processor and just write to the database for each record this error does not occurr, but writing for each object is very slow. Why does this happen? How can I debug this? ... View more