NIFI - How can I validate record?

[#1]

Hello, I have to read a text file but I need to discard invalid records. For example I need to discard the records with a invalid column date '-190382734' but only I need a good dates records for example '09202018'. it's posible with NIFI?

1.jpg

Thanks!!

[#2]

It's totally possible with Nifi!

There are a number of ways you could go about it, and it depends on if you think the dates will have the same format, or if they will vary. (ie: mm/dd/yyyy or dd-mm-yyyy)

I found another useful conversation here that covers using the routeOnContent or routeOnText processors to evaluate a regex expression, checking if the value matches what you expect.

I would recommend looking at the documentation on those processors, as well as testing out your regex expression on a site like Regexr. Hope that gets you started!

[#3]

You should be able to use ValidateRecord for this, although your reader schema might need to treat the incoming field as an int/long so it will read in negative numbers successfully. Then your write schema can have that field with its correct type (date, e.g.)

You can also use QueryRecord for this, it allows you to query the flow file fields using SQL, such as "SELECT * FROM FLOWFILE WHERE columnDate >= 0" or whatever is appropriate.