Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Replace Text using Regex with large file in NIFI

avatar
Expert Contributor

Hi,

I'm using HDP 2.5 with Nifi 0.6 version. I have 10GB CSV file, i want replace double quotes with pipe delimiter. Is it possible on NIFI Replace Text processor? or I need to write any external script like Groovy,Luna. Please tell me any reference links, how to implement either scripting or replace text processor for large file.

1 ACCEPTED SOLUTION

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
8 REPLIES 8

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

Thanks for response. I will try this.I have regex like 's/","/\|/g; s/"//g', How do i use in this replace text processor.

avatar
Expert Contributor

It works!!!!

avatar

Hello,

Anshu here.

We have a requirement to anonymize IP address. So we would identify IP address and replace the last part with some arbitrary value.

We have tried using the following regex for that (with "Regex Replace" for replacement strategy, "Line-by-Line" as evaluation mode)

(1) \\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}

(2) ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$

(3) ((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)

None of this works.

Can you help me with the right value of regex we can use and the replacement value as well.

Thank you in advance!

avatar
Super Mentor

@Anshuman Ghosh

Please open a new question for this issue rather then attaching to a question with an existing answer. While dealing with the same processor, the problem is different and it benefits the community to have it addressed under its own question.

Thanks,

Matt

avatar
Super Mentor

@Anshuman Ghosh

Tip: Make sure you tag your question with NiFi, so that the community of NiFi followers are notified about your new question.

avatar

@Matt Clarke

Thanks! I will do that. Regret for any inconvenience.

avatar
Super Mentor

@Anshuman Ghosh

No inconvenience at all. We just want to keep it as easy as possible for community members to find similar issues and solutions. I would have moved it myself if I couldhave. 🙂