Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to fetch header & trailer records from a flowfile in NiFi?

Solved Go to solution
Highlighted

How to fetch header & trailer records from a flowfile in NiFi?

I have the following records in my file,

0$20011227$20011228$1 1$Suresh 1$Kaamesh 1$Vijay 1$Harish 1$Sabareesh 1$Vaneesh 2$5$1$2$100

The record which starts with '0' is the header record, '1' are the detail records & '2' is the footer record. I need to fetch the header & trailer/footer record to do some validations.

Could you please let me know how this can be achieved in NiFi using regex or some other way?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to fetch header & trailer records from a flowfile in NiFi?

The simplest way to do this would be an ExtractTest processor. You could configure it like this:

Then you will have the header and footer records in attribute fields called record_header and record_footer, respectively:

View solution in original post

8 REPLIES 8
Highlighted

Re: How to fetch header & trailer records from a flowfile in NiFi?

New Contributor

Is each record ended with a newline? Or is it just a long list of records?

Highlighted

Re: How to fetch header & trailer records from a flowfile in NiFi?

Hi David, the lines are terminated by '\n' at the end.

Highlighted

Re: How to fetch header & trailer records from a flowfile in NiFi?

The simplest way to do this would be an ExtractTest processor. You could configure it like this:

Then you will have the header and footer records in attribute fields called record_header and record_footer, respectively:

View solution in original post

Highlighted

Re: How to fetch header & trailer records from a flowfile in NiFi?

Hi Hellmar,

Thanks for your response, it worked!!

I also tried using the below regex by setting the Multi-line mode to 'True' in the ExtractText processor,

Header - ^[0](.*) Footer - ^[2](.*)

I got the same result. Do you see any issues with this approach?

Re: How to fetch header & trailer records from a flowfile in NiFi?

Contributor

@Rohit Ravishankar

That should be fine, you could also use splitText processor before the extractText processor. So basically each flowfile would contain a single record.

Highlighted

Re: How to fetch header & trailer records from a flowfile in NiFi?

Thanks Devin!!!

Highlighted

Re: How to fetch header & trailer records from a flowfile in NiFi?

No issues - usually with regexes, there are different ways to reach the same goal.

Highlighted

Re: How to fetch header & trailer records from a flowfile in NiFi?

Thanks Hellmar!!!

Don't have an account?
Coming from Hortonworks? Activate your account here