Nifi RouteOnAttribute Problem

[#1]

Hi Does anyone know if it is by design that Nifi RouteOnAttribute on look at an attribute value that is for the entire flowfile instead of attribute value that can be different for different record in a josn file? e.g., when RouteOnAttribute is set to find a word 'xyz' in a json element (say msg), e.g, properties configure to route on ${msg:find('xyz')}, it actually route entire file including records that msg does not contain xyz. If that how it works or if the RouteonAttribute properties configuration is missing something?

Thanks,

Shannon

Update: Edit to add additional information based on @Chris Gambino and @jwitt replies:

Data flow is design as:

GetSourceData --> some process for parsing to get a flat json --> EvaluateJsonPath--> UpdateAttribute -- > RouteOnAttribute

May be the problem is at UpdateAttribute, which has a property added to an attribute "matched_msg" using below so that this new attribute has 1 or 0 value.

${msg:find('xyz'):equals('true'):replace('true', '1'):or (${tmsg:find('xyz'):equals('false'):replace('false', '0')})}

Above followed by RouteOnAttribute using ${matched_msg:equals(1)}

Results is that it ended up with entire file instead of records with the match.

[#2 · Accepted solution]

Hello,

Keep in mind each FlowFile is comprised of attributes and content.

RouteOnAttribute is built specifically to look at FlowFile attributes.

RouteOnContent looks at FlowFile content.

In the case of JSON you can, for example, use EvaluateJSONPath to extract fields from the JSON to become FlowFile attributes. You can use SplitJSON to split the JSON bundles into invidual flowfiles then extract values and then route on attributes. The various processors offer you a variety of ways to route, transform, and deliver the data.

Thanks Joe

[#3]

@jwitt, @Chris Gambino

Would you be so kind to take a look at updated info above (the second part of the original posting)? I tried to simply the description in the beginning, which was a wrong call :). May be the problem is not in the processor I originally thought. May be UpdateAttribute was not configured right with the expression given? In the output file, I can't see the new attribute match_msg added, which was not what I expected either. Any ideas?

Thanks!

Shannon

[#4]

Update: if just connect EvaluateJsonPath to RouteOnAttribute, configure it to route on ${msg:find('xyz'):equals('true')}. The result is still the same-- getting entire file instead of the records has xyz in msg. So, guess it is still RouteOnAttribbute is having problem. Any help on correcting it would be greatly appreciated.

Thanks,

Shannon

[#5]

@jwitt

yep-- this is the solution, EvaluateJsonPath only deals with one record at a time not a whole array. Thanks much for pointing to the right direction! Shannon

[#6]

What is the JSON key of the value you are trying to extract?

[#7]

Hey Shannon,

You want to use EvaluateJsonPath to make an attribute called "msg" with the value $.msg

You then want to use RouteonAttribute with ${msg:equals('xyz')}

You are very close to having it down correctly!

[#8]

Hey @shannon luo just wanted to chime in. In your data flow design you mentioned a step where you're parsing data to get a flat json. In case that process is changing the schema of json wanted to just let you know there's an upcoming processor called TransformJSON that may be useful. It supports Jolt specifications (https://github.com/bazaarvoice/jolt) and makes it a lot easier to transform or extract json flowfile content. If that's not what you need now it may be helpful in the future working with json in NiFi.

[#9]

@Yolanda M. Davis

Great to know. Thank you, Yolanda!