Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to filter the incoming JSON payload based on the values in flow file content using NiFi?

How to filter the incoming JSON payload based on the values in flow file content using NiFi?

New Contributor

Hi,

I have the incoming flowfiles with the json content which have multiple event types.

I need to filter only certain events based on two of the JSON key/value.

Sample input data:

{"eventType":"Mobile","timestamp":1551280374552,"name":"ERROR"}
{"eventType":"Mobile","timestamp":1551280374552,"name":"APP_START"}
{"eventType":"Immobile","timestamp":1551280374552,"name":"ERROR"}
{"eventType":"Immobile","timestamp":1551280374552,"name":"ERROR"}
{"eventType":"Mobile","timestamp":1551280374552,"name":"PLAYBACK_ERROR"}
{"eventType":"Mobile","timestamp":1551280374552,"name":"Other"}
{"eventType":"MobileCrash","timestamp":1551280374552,"name":"ERROR"}

Filter Condition to be applied :

Flowfile content having

If eventtype = "Mobile" and name = (ERROR or APP_START or PLAYBACK_ERROR) or

eventtype = "MobileCrash"


Also, after the filter i have to rename the json key as,

eventType = vendorEventType

timestamp = currentTimestamp

name = someName


Desired Output:

{"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"ERROR"}
{"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"APP_START"}{"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"PLAYBACK_ERROR"}{"vendorEventType":"MobileCrash","currentTimestamp":1551280374552,"someName":"ERROR"}


Please help.


1 REPLY 1

Re: How to filter the incoming JSON payload based on the values in flow file content using NiFi?

Super Guru

@Bala S

Use QueryRecord processor with Record Reader and Record Writer controller services, and QueryRecord processor will result out the records that matches your Query.

Use ApacheCalcite sql syntax to filter out the required records from the flowfile content.

Some useful links link1, link2