question Re: ReplaceTextWithMapping processor. De-duplicate only specific columns in a file in Archives of Support Questions (Read Only)

ReplaceTextWithMapping processor. De-duplicate only specific columns in a file

rbalakrishnantc — Tue, 15 Nov 2016 18:26:30 GMT

I have a CSV file with 9 columns. How can I remove duplicates among columns 4 through 9?

What we tried:

1. Split 1-4 columns in a file

2. Split 4-9 columns -> Deduplicate records

Now, i tried using 'ReplaceTextWithMapping' to merge the files with 4th column (Common on both files). But I am not sure if my approach is right.

Is there any other way to achieve this.

Re: ReplaceTextWithMapping processor. De-duplicate only specific columns in a file

pvillard — Wed, 16 Nov 2016 01:09:15 GMT

Hi @bala krishnan ,

Not a solution but just to let you know that with the next version of NiFi (coming soon) you will be able to use ValidateCSV processor to achieve what you are looking for. In the meantime, I think that splitting the file is not going to help. Maybe trying something custom with ExecuteScript processor but probably not ideal.

Hope this helps.