question Re: How do I search through FlowFiles and pull out all lines that match a multiple string values? in Support Questions

How do I search through FlowFiles and pull out all lines that match a multiple string values?

MPHSpeed — Wed, 17 Jan 2024 17:39:48 GMT

Greetings,

I'm currently grabbing data from a website, "splitting" it into its separate components and trying to route them so I can save them into a single zipped file.

My workflow currently works as follows:
InvokeHTTP --> SplitText --> RouteText ...

The original Dataset is a bunch of satellite data (example below). I’m then splitting it into separate 3 line records. I then run RouteText and try and grab specific records. For example, if I wanted TDRS 3 and AMSC 1 and SKYNET 4C, how do I pull them out with its respected 2 lines? I tried adding a Property: SatelliteData1 = ${literal('TDRS 3')}. This only gives me the first line of the Split record (I need all 3). I tried OR-Statement, but don't think I'm using it right.

${literal('TDRS 3''):or(${literal('AMSC 1')}):or(${literal('SKYNET 4C')})}

Any suggestions will be greatly appreciated. Thanks in advance.

Original Data:

Splitting:

Re: How do I search through FlowFiles and pull out all lines that match a multiple string values?

DianaTorres — Wed, 17 Jan 2024 17:47:14 GMT

@MPHSpeed Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @steven-matison who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: How do I search through FlowFiles and pull out all lines that match a multiple string values?

MattWho — Wed, 17 Jan 2024 21:37:34 GMT

@MPHSpeed

Rather than using RouteText processor which routes individual lines of a text file, you could use RouteOnContent processor that routes the entire FlowFile whose content matches to a dynamic relationship.

What i would do is extract the data type TDRS 3, or AMSC 1 or SKYNET 4C, etc to a FlowFile attribute using ExtractText processor and then you have that type associated with the FlowFile through your entire flow making it easy to do things like merge FlowFiles all of same type together (MergeContent with "Correlation Attribute Name"), route FlowFiles of a specific type using RouteOnAttribute, etc...

Then you also have options using the many Record based processors if you can define a schema for your data that defines your record as those three lines.

Example:
SplitText (splits relationship) ---> ExtractTEXT:

ExtractText (Matched relationship) --> RouteOnAttribute

RouteOnAttribute with above configuration will have three dynamically created relationships for the data types you want to keep. connect each to the unique dataflow path for processing that data type.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: How do I search through FlowFiles and pull out all lines that match a multiple string values?

MPHSpeed — Wed, 17 Jan 2024 23:03:41 GMT

@MattWho I get what we are doing. It makes sense and seems to be the right way to handle it. Unfortunately, It seems to send everything to 'unmatched' on ExtractText (at least on my end). I made sure I had everything the same as what you provided.
I'm extracting data from here:
https://celestrak.org/NORAD/elements/gp.php?GROUP=geo&FORMAT=tle

SplitText:

ExtractText:

RouteOnAttribute:

Re: How do I search through FlowFiles and pull out all lines that match a multiple string values?

MattWho — Thu, 18 Jan 2024 17:17:21 GMT

@MPHSpeed

Working with the actual data instead of sample data I build, i would recommend making these two changes:

1. In extractText processor change "Enable Unix Lines Mode" to true.
2. In each dynamic property in RouteOnAttribute, change "equals" function to "contains" function.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt