To loop through the dataflow

[#1]

I have dataflow files which has to be routed on the basis of certain criteria.

It has multiple field like ("Customefied_23456: "Null") in the key: value pair. My query is how  to filter out the flowfile which contain the some value for customfield ("customfield_33749": "this is required value",). i.e it does not contain null like  ("Customefied_23456: "Null").

Please let me know how to proceed in this case.

thanks in advance.

[#2]

@MattWho 

Please advise me on this issue.

Issue:

 In the flow file which contain at least customfield_  with some value other than null should be routed to another processor and the flowfile which has all null value should be moved to other processor.

thanks in advance!

[#3]

@midee 

This use case is really not clear to me.

The image you shared is the content of a single FlowFile and that content has numerous "customfield_" fields with most being "null" and one having a string value.

So you are asking that this 1 FlowFile with both null and non-null "customfield_" fields is routed to the path A because at least one "customfield_" field as a non-null string?  The content would remain unedited.

And you want other FlowFiles where the content contains nothing but all "customfield_" fields with null value routed to path B?  The content would remain unedited.

Thanks,

Matt

[#4]

@MattWho ,

Yes, your analysis is correct.

So you are asking that this 1 FlowFile with both null and non-null "customfield_" fields is routed to the path A because at least one "customfield_" field as a non-null string?  The content would remain unedited.  ..Yes, the flowfile which will contain at least one non-null value should be moved to other processor without content modification.

And you want other FlowFiles where the content contains nothing but all "customfield_" fields with null value routed to path B?  The content would remain unedited. --Yes, it is the same.

Please suggest on this.

Thanks!

[#5 · Accepted solution]

@midee 

You could use a routeOnContent [1]processor to accomplish this.  You would create a Java regex that matches only on customfield(s) where there is a string wrapped in quotes.  If found, it routes entire FlowFile to the relationship created using the dynamic property's name.

RouteOnContent configuration:

I noticed in you example you have two customfield entries that do not have "null"

"customfield_10001": "This is required value",
"customfield_10002": "",

Based on my regex provided above, both of these would match resulting in FlowFile being routed to the "NotNull" relationship.
If i were to change the second .*? to .+? , then the customfield that contained only quotes would not match (just in case you only want to route when it is not null and not empty.

"customfield_.+?": ".*?",

versus

"customfield_.+?": ".+?",

If you found this addressed your query, please take a moment to login and click "accept" on this solution.
Thank you,

Matt

[#6]

@MattWho ,

Thanks for your response and it work as per my requirement.

thanks!

[#7]

@MattWho ,

When I applied the solution provided by you on the flowfile (formatted json ), it works as expected.

However, when the same expression was applied to the flowfile in original form(one line json file), it was not working as expected.

Could you please let me know this unexpected behavior.

[#8]

@midee 

I just ran the test myself with your sample as a one line formatted json and it still worked.

{"expand": "renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id": "1155391","key": "DPGAUT-2","fields": {"customfield_10001": null,"customfield_10002": null,"customfield_10003": null,"customfield_10004": null,"customfield_10005": null,"customfield_10006": null,"resolution": null,"customfield_10101": "","customfield_10007": null,"customfield_10008": null,"customfield_10009": "This is required value","customfield_10010": null,"customfield_10011": null,"customfield_10012": null,"customfield_10013": null,}}

Make sure your RouteOnContent processor dynamic property value does not have a line return in it.

You'll notice it only shows "1" line above. If you see a "2" line then you have a line return in your Value field which would cause the issue you are seeing.

Hope this helped,

Matt