Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NIFI - How can I validate record?

Highlighted

NIFI - How can I validate record?

New Contributor

Hello, I have to read a text file but I need to discard invalid records. For example I need to discard the records with a invalid column date '-190382734' but only I need a good dates records for example '09202018'. it's posible with NIFI?

1.jpg

Thanks!!

2 REPLIES 2

Re: NIFI - How can I validate record?

New Contributor

It's totally possible with Nifi!

There are a number of ways you could go about it, and it depends on if you think the dates will have the same format, or if they will vary. (ie: mm/dd/yyyy or dd-mm-yyyy)

I found another useful conversation here that covers using the routeOnContent or routeOnText processors to evaluate a regex expression, checking if the value matches what you expect.

I would recommend looking at the documentation on those processors, as well as testing out your regex expression on a site like Regexr. Hope that gets you started!

Re: NIFI - How can I validate record?

You should be able to use ValidateRecord for this, although your reader schema might need to treat the incoming field as an int/long so it will read in negative numbers successfully. Then your write schema can have that field with its correct type (date, e.g.)

You can also use QueryRecord for this, it allows you to query the flow file fields using SQL, such as "SELECT * FROM FLOWFILE WHERE columnDate >= 0" or whatever is appropriate.