question Re: How to remove ETX character from json value ? in Support Questions

How to remove ETX character from json value ?

Ghilani — Wed, 09 Nov 2022 14:05:47 GMT

Hello,

I could not split a JSON due to a special ETX character existing in a JSON value, SplitJson processor returns an error that it is not a Valid JSON.

The JSON looks like that:
[ {"key_data":"val_data"},

{"key_data":"val_ETXdata"},

{"key_data":"val_data"},

{"key_data":"val_data"},... ]

I used The next regex expression to replace it but it doesn't work:

\x03(?=[^"]*")

also

[\x03](?=[^"]*")

can someone show me a trick to remove it please, I would appreciate it.

Re: How to remove ETX character from json value ?

SAMSAL — Wed, 09 Nov 2022 16:37:23 GMT

Hi,

I think you are having an issue because you have carriage return (\r\n) in the json , try using regex replace for the following as well: [\r\n]

Hope that helps, if it does please accept solution.

Thanks

Re: How to remove ETX character from json value ?

Ghilani — Wed, 09 Nov 2022 16:42:09 GMT

No, it is not the problem, the problem is due to the EXT symbol, \n\r\t\" am filtering them in other processors, Am filtering more than 10000 JSON files the error happens only with 4 files that contain ETX.

Re: How to remove ETX character from json value ?

SAMSAL — Wed, 09 Nov 2022 21:09:06 GMT

can you send me a sample json data with the error. The one you posted seems to be valid and Im able to split it .

Re: How to remove ETX character from json value ?

Ghilani — Thu, 10 Nov 2022 10:22:14 GMT

share please your email

Re: How to remove ETX character from json value ?

SAMSAL — Thu, 10 Nov 2022 16:23:46 GMT

Hi,

I tried the follow pattern on the sent file and it worked: [\x03]+

Re: How to remove ETX character from json value ?

Ghilani — Thu, 10 Nov 2022 16:36:36 GMT

I tried both solutions
I got this error

Failed to process session due to null; Processor Administratively Yielded for 1 sec: java.nio.BufferOverflowException

Re: How to remove ETX character from json value ?

SAMSAL — Thu, 10 Nov 2022 19:27:01 GMT

can you share the configuration for the ReplaceText Processor? Also how big is the jsonfile?

Re: How to remove ETX character from json value ?

Ghilani — Thu, 10 Nov 2022 19:31:57 GMT

properties are the same as in the picture you have shared.

Re: How to remove ETX character from json value ?

SAMSAL — Thu, 10 Nov 2022 20:08:58 GMT

Not sure how big is your Json and if its well formatted into multiple line. Make sure you have the Evaluation Mode is set to Line-by-Line , also you can increase the Maximum Buffer Size incase the text processed is greater than 1MB. Also what version of Nifi are you using ? there seems to be a bug around that as well where the flowfile will remain in the upstream queue and the overflow error is thrown: https://issues.apache.org/jira/browse/NIFI-10154

Re: How to remove ETX character from json value ?

Ghilani — Thu, 10 Nov 2022 20:17:37 GMT

I think everything works fine, the json flowfiles are the response of an invokeHttp request to an API, I receive more than 5000 flowfiles, each flowfile contains 200 records.
Problems happen only with 4 files that contain the ETX symbol.