[APACHE NIFI] How to extract only the last record from a csv file

[#1]

I have a csv file which a new record comes and appends every 15 minutes. I need to extract only the latest record from the csv file and save as a JSON

[#2]

@bhadraka, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @cotopaul @SAMSAL  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

I have managed to get the csv file and converting it into a JSON but my JSON file contains all the data, I want to filter only the last record from the csv and convert only that record into a JSON

[#4]

@bhadraka,

Out of the box, I would try the following two scenarios, to see which one fits your use case better:

1) I would use Tail File to monitor that CSV File. I assume that all the newly added rows are added to the end of the file so once your processor gets executed, the TailFile will output either one line per flowfile or multiple lines per flowfile, depending on your schedule.

2) I would use FetchFile-ListFile or GetFile and extract the CSV File, without deleting the file. I will then proceed to split the records using SplitRecord and only process the last row, using an RouteOnAttribute based on fragment index.

[#5 · Accepted solution]

@bhadraka 

What version of NiFi are you using? In NiFi 1.20.0, you can use ReplaceText Processor after reading in the file. 

Using the line-by-line evaluation mode, there is a drop down "Except-Last-Line". You could then configure it to just replace all previous lines with empty strings.

Here's a screenshot of my ReplaceText processor properties.