question Re: To loop through the dataflow in Support Questions

To loop through the dataflow

midee — Sun, 06 Jun 2021 14:59:55 GMT

I have dataflow files which has to be routed on the basis of certain criteria.

It has multiple field like ("Customefied_23456: "Null") in the key: value pair. My query is how to filter out the flowfile which contain the some value for customfield ("customfield_33749": "this is required value",). i.e it does not contain null like ("Customefied_23456: "Null").

Please let me know how to proceed in this case.

thanks in advance.

Re: To loop through the dataflow

midee — Tue, 08 Jun 2021 14:47:35 GMT

@MattWho

Please advise me on this issue.

Issue:

In the flow file which contain at least customfield_ with some value other than null should be routed to another processor and the flowfile which has all null value should be moved to other processor.

thanks in advance!

Re: To loop through the dataflow

MattWho — Wed, 09 Jun 2021 13:33:13 GMT

@midee

This use case is really not clear to me.

The image you shared is the content of a single FlowFile and that content has numerous "customfield_" fields with most being "null" and one having a string value.

So you are asking that this 1 FlowFile with both null and non-null "customfield_" fields is routed to the path A because at least one "customfield_" field as a non-null string? The content would remain unedited.

And you want other FlowFiles where the content contains nothing but all "customfield_" fields with null value routed to path B? The content would remain unedited.

Thanks,

Matt

Re: To loop through the dataflow

midee — Wed, 09 Jun 2021 13:49:42 GMT

@MattWho ,

Yes, your analysis is correct.

So you are asking that this 1 FlowFile with both null and non-null "customfield_" fields is routed to the path A because at least one "customfield_" field as a non-null string? The content would remain unedited. ..Yes, the flowfile which will contain at least one non-null value should be moved to other processor without content modification.

And you want other FlowFiles where the content contains nothing but all "customfield_" fields with null value routed to path B? The content would remain unedited. --Yes, it is the same.

Please suggest on this.

Thanks!

Re: To loop through the dataflow

MattWho — Mon, 14 Jun 2021 13:03:01 GMT

@midee

You could use a routeOnContent [1]processor to accomplish this. You would create a Java regex that matches only on customfield(s) where there is a string wrapped in quotes. If found, it routes entire FlowFile to the relationship created using the dynamic property's name.

RouteOnContent configuration:

I noticed in you example you have two customfield entries that do not have "null"

"customfield_10001": "This is required value", "customfield_10002": "",

Based on my regex provided above, both of these would match resulting in FlowFile being routed to the "NotNull" relationship.
If i were to change the second .*? to .+? , then the customfield that contained only quotes would not match (just in case you only want to route when it is not null and not empty.

"customfield_.+?": ".*?",

versus

"customfield_.+?": ".+?",

If you found this addressed your query, please take a moment to login and click "accept" on this solution.
Thank you,

Matt

Re: To loop through the dataflow

midee — Thu, 17 Jun 2021 06:59:33 GMT

@MattWho ,

Thanks for your response and it work as per my requirement.

thanks!

Re: To loop through the dataflow

midee — Tue, 22 Jun 2021 12:45:54 GMT

@MattWho ,

When I applied the solution provided by you on the flowfile (formatted json ), it works as expected.

However, when the same expression was applied to the flowfile in original form(one line json file), it was not working as expected.

Could you please let me know this unexpected behavior.

Re: To loop through the dataflow

MattWho — Thu, 24 Jun 2021 12:34:13 GMT

@midee

I just ran the test myself with your sample as a one line formatted json and it still worked.

{"expand": "renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id": "1155391","key": "DPGAUT-2","fields": {"customfield_10001": null,"customfield_10002": null,"customfield_10003": null,"customfield_10004": null,"customfield_10005": null,"customfield_10006": null,"resolution": null,"customfield_10101": "","customfield_10007": null,"customfield_10008": null,"customfield_10009": "This is required value","customfield_10010": null,"customfield_10011": null,"customfield_10012": null,"customfield_10013": null,}}

Make sure your RouteOnContent processor dynamic property value does not have a line return in it.

You'll notice it only shows "1" line above. If you see a "2" line then you have a line return in your Value field which would cause the issue you are seeing.

Hope this helped,

Matt