Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NIFI - remove header and footer lines from CSV

Highlighted

NIFI - remove header and footer lines from CSV

New Contributor

I am attempting to remove multiple header and footer/trailer lines from a CSV.  The CSVReader will only allow me to skip the first line, which isn't enough,and I don't see any way to skip trailer records.  Is my only option to use an ExternalScript processor and handle it outside of nifi? Like a shell script with head and tail?

 

2 REPLIES 2
Highlighted

Re: NIFI - remove header and footer lines from CSV

Master Collaborator

@JohnYaya  if you can show a sample, it would be very helpful.

 

If your header and footer is static, and always predictable for a specific file you can replaceText and get "in between" lines using those static header/footer matches with a very creative regex...

 

 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: NIFI - remove header and footer lines from CSV

New Contributor

Well, I'm trying to put together a fairly generic service to ingest files.  One of the reqs is to be able to strip off any header or trailer lines.  Files can come from any number of sources, so I don't think there will be any kind of pattern for a regex.  

Don't have an account?
Coming from Hortonworks? Activate your account here