Detecting hidden characters NiFi

[#1]

Is there a processor that can detect hidden characters in NiFi?

In our test environment I am using an API invoke to deliver a JSON payload, for added security we want to filter out any hidden and potentially malicious text.

I have been trying to use RouteText, RouteOnContent and ExtractText processor to only allow alphanumeric and punctuation characters through but can't seem to get the regex to work when looking for 'uncommon' text or characters?

any help would be appreciated.

Andy 

[#2 · Accepted solution]

Regex remove those bad characters

https://community.cloudera.com/t5/Support-Questions/nifi-regex-replace-special-characters/td-p/10340...

https://community.cloudera.com/t5/Support-Questions/How-do-I-enter-an-unprintable-byte-into-a-nifi-p...

https://community.cloudera.com/t5/Support-Questions/Remove-from-a-flow-file-in-Nifi/td-p/109503

https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html

http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-and-special-characters-td480.html

https://community.cloudera.com/t5/Support-Questions/ReplaceText-quot-processor-does-not-replace-spec...

https://community.cloudera.com/t5/Support-Questions/remove-special-characters-from-xml-text-node-usi...

https://community.cloudera.com/t5/Support-Questions/Regex-Special-Character-Escape/m-p/239556#M20136... 

Could also use UpdateRecord on Json with infer with replace or replaceregex

https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html

[#3 · Accepted solution]

Regex remove those bad characters

https://community.cloudera.com/t5/Support-Questions/nifi-regex-replace-special-characters/td-p/10340...

https://community.cloudera.com/t5/Support-Questions/How-do-I-enter-an-unprintable-byte-into-a-nifi-p...

https://community.cloudera.com/t5/Support-Questions/Remove-from-a-flow-file-in-Nifi/td-p/109503

https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html

http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-and-special-characters-td480.html

https://community.cloudera.com/t5/Support-Questions/ReplaceText-quot-processor-does-not-replace-spec...

https://community.cloudera.com/t5/Support-Questions/remove-special-characters-from-xml-text-node-usi...

https://community.cloudera.com/t5/Support-Questions/Regex-Special-Character-Escape/m-p/239556#M20136... 

Could also use UpdateRecord on Json with infer with replace or replaceregex

https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html

[#4 · Accepted solution]

@Griggsy 

Here is an example of how you could use ReplaceText:

Search value:

[^\w\d\r\n! @^$*#()_=<>~`|{}:;,.\-\\\?\/]+

The below site is a great way to test above regex against some sample data you have to make sure you are not missing any exceptions you want to keep
https://regex101.com/

It also explains this above regex formatting

If you found these responses helpful, please take a moment to login and click "Accept" on all solutions that help you.

Thanks,

Matt

[#5]

Thanks for both replies, I managed to get it working last week the same way as you have shown Matt.

Cheers