question Remove first few lines in a text/csv flowfile content in Apache NiFi in Support Questions

Remove first few lines in a text/csv flowfile content in Apache NiFi

glad1 — Wed, 22 Nov 2023 09:06:48 GMT

I receive a text/csv file with many lines through an InvokeHttp Processor. My requirement is that I don't want the first 7 lines. What should I do to remove the first 7 lines and keep the remaining the same text/csv format.

Re: Remove first few lines in a text/csv flowfile content in Apache NiFi

SAMSAL — Wed, 22 Nov 2023 17:23:00 GMT

Hi @glad1 ,

Can you elaborate more on the data that you want to remove? For example if the data is part of the CSV and it has unique value in one or more columns, then you can use QueryRecord processor where the query exclude records with this unique value. If the data is out of the CSV - like a header information - then depending how this data look like and if its surrounded with some special characters then you can use ReplaceText Processor with regex that would isolate those lines and then replace them with empty space and so on. If you can provide some sample data it would help in figuring out the best solution for this scenario.

Thanks

Re: Remove first few lines in a text/csv flowfile content in Apache NiFi

joseomjr — Wed, 22 Nov 2023 19:33:36 GMT

I you're confident the data returned is consistent and always more than 7 lines...then a quick and dirty would be a Groovy script like this.

import java.nio.charset.StandardCharsets FlowFile flowFile = session.get() if(!flowFile) return flowFile = session.write(flowFile, {inputStream, outputStream -> String[] data = inputStream.readLines() data = data.drop(7) outputStream.write(data.join("\n").getBytes(StandardCharsets.UTF_8)) } as StreamCallback) session.transfer(flowFile, REL_SUCCESS)

Re: Remove first few lines in a text/csv flowfile content in Apache NiFi

glad1 — Thu, 23 Nov 2023 05:49:47 GMT

Thank you, this worked!

Re: Remove first few lines in a text/csv flowfile content in Apache NiFi

glad1 — Thu, 23 Nov 2023 05:52:57 GMT

^ I've attached the image above. this is how the data looks. I want to clean the first 7 rows and let the 8th row (header row) be first.