Cloudera Community
michael_strasse
user rank
New Contributor

Re: Validate input file row count in Nifi?

by New Contributor in Support Questions
‎05-17-2016 06:53 AM
‎05-17-2016 06:53 AM
Hi @Pierre Villard Thanks for your quick response and the link to Matt Burgess's blog. I think that ExecuteScript will be a good way to go in this case. I have also been told that we might not reject the whole file if the row count is incorrect. Regarding the linkage between whole files and split flow files, downstream of this process the data will be enriched in some way with information about its source, perhaps via the segment.original.filename property of SplitText. ... View more

Validate input file row count in Nifi?

by New Contributor in Support Questions
‎05-17-2016 05:20 AM
‎05-17-2016 05:20 AM
I have a data ingestion scenario that I am trying to implement in Apache Nifi. The input is a delimited file with a header and a footer. A trivial example is: itno|col1|col2|col3 20123456|10|50|10 20434561|20|0|20 20342345|10|10|20 F|3 The header contains column names and the footer contains F followed by the row count. I want to create a Process Group to do the following with the entire delimited file as input FlowFile: discard the header split the rows into separate FlowFiles validate that the row count is correct and reject the entire file if it is wrong This means that I don't want to emit rows for further processing until all rows are read and the count validated. Is there an efficient way to do this? ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎05-17-2016 06:53 AM
Community Statistics
Member Since ‎05-17-2016 02:18 AM
Last Visited ‎05-17-2016 06:53 AM
Posts 2