NiFi - Replace Text IndexOutOfBoundsException

[#1]

Hi,

I am extracting a big table and I am using a processor Replace Text but I am getting the error :

The processor is configured as follow : (I extended Maximum Buffer Size to 100 MB)

Any idea why I am getting this error?

[#2]

Could you provide your entire replacement value and the event payload? It looks to me that your event does not include one of the attributes/parameters in capture group 5.

[#3]

@bhagan

I just changed the replacement value to : "${id:replaceEmpty('')}"||"${social_id:replaceEmpty('')}"||"${name:replaceEmpty('')}"||"${screenName:replaceEmpty('')}"||"${location:replaceEmpty('')}"

An event payload would be (JSON format) :

{"id":"1759930","social_id":"116161419","name":"Lucy Lloyd","screenName":"lucy_lloyd1","location":"Tunbridge Wells, England ","bio":"","lang":"en","created_date":"2010-02-21 11:35:07.0","image_url":"http://pbs.twimg.com/profile_images/787703929786232833/FRoV7x0L_normal.jpg","backgroud_image":"http://pbs.twimg.com/profile_background_images/600155425/xgqomh222q1tiq387bch.jpeg","followers":1011,"friends":862,"status":11429,"timezone":"London","url":null,"poids_social":23.98221972589741,"age":null,"gender":null,"smile":null,"glasses":null,"moustache":null,"beard":null,"sideburns":null,"country":null,"source":"LOCATION"}

I am trying to convert a json to csv format.

I guess that the fact to add double quotes "" in the replacement valued helped to solve the problem.

I think there was special characters in group 5 (like the character $).

Using double quotes happens to solve my problem but is there another solution to handle special characters like $ and carriage return? cause some of my fields have some of them.

[#4]

@Raphaël MARY

Are you splitting the JSON into separate flowfiles, so that each JSON element (row of your data) is being processed individually by the ReplaceText processor? Take a look at two StackOverflow answers that I found dealing with handling multiline data:

http://stackoverflow.com/a/3652392

http://stackoverflow.com/a/17825571

It looks like you may still need to use (?s) to turn on DOTALL mode so that EOL characters are matched/consumed by your regex expression. If ReplaceText is looking at a single JSON element (row of data), DOTALL mode will make the "." character consume EOL characters.

For further information, here's a link that contains documentation on the match flags (such as the "s" flag in (?s)): https://docs.oracle.com/javase/tutorial/essential/regex/pattern.html

I'm not a regex expert by any means, but hopefully some of this information helps you!