question Parsing flow file or attribute in Nifi in Support Questions

Parsing flow file or attribute in Nifi

anilcseiitm2006 — Thu, 11 May 2017 03:58:12 GMT

I have two scenario's where I need help to extract the values

1) Flow file have the content as below

*********content of flow file***********

field_1=field1value&feild_2=field2value&field_3=field3value&...&fieild_n=fieldnvalue

********End of Content***************

I want to extract values of field1, field2, field3 ... field_n and store them to 3 attributes. Can I get regular expression example to do that using ExtractText in Nifi.

or say I want to extract value of field_x (1 < x < n) attribute from above list, how can I do that ?

2) If I have an attribute with value as below

**********attrbitue vlaue***********

field_1=field1value&feild_2=field2value&field_3=field3value&...&fieild_n=fieldnvalue

***********************************

I want to extract value of field_x (1 < x < n) attribute from above list, how can I do that ?

Re: Parsing flow file or attribute in Nifi

ssahi — Thu, 11 May 2017 12:31:25 GMT

Hi @Anil Reddy

If you are like me, and dislike RegEx, one trick you can try is to use the SplitContent processor first. Change config dropdown to use Text instead of Hexadecimal, and use the byte sequence of your pair delimiter &. This would simplify the RegEx if you wanted to use ExtractText still. Or perhaps you can explore using another SplitContent processor on the = to get the field and value tokens separately. Hopefully you can avoid the RegEx there.

As always, if you find this post helpful, please accept the answer.

Re: Parsing flow file or attribute in Nifi

MattWho — Sun, 18 Aug 2019 01:18:02 GMT

@Anil Reddy

You could use extractText processor configured similar to the following:

I changed the two shown standard properties and added two additional regex properties.

Using the following example input:

field_1=field1value&field_2=field2value&field_3=field3value&field_4=field4value

You will end up with the following attributes on your FlowFile:

There will be a few additional attributes created that you can ignore, but you will have a sequentially numbered attribute names with the associated values and one field_last that will have the very last value in your input string.

Thanks,

Matt

Re: Parsing flow file or attribute in Nifi

anilcseiitm2006 — Fri, 12 May 2017 00:55:10 GMT

@Matt Clarke: thanks for the solution and it works specifically to the example of field_1, field_2.

Actually my intention is, I have couple of fields as part of flow file which are separated by character &. My requirement is I would like to extract the specific field based on field name. I have been trying to achieve that experimenting regular expressions but could not able to succeed so far. Infact I ma trying to understand the regular expressions you specified but could not get hold of it.

can you please let me know how to extract the fields in the flow file whose content is say

firstname=testfirstname&lastname=testlastname,email=testemail@email.com,address=test address.

I want to extract attributes firstname, lastname, email, address with appropriate values.

Re: Parsing flow file or attribute in Nifi

anilcseiitm2006 — Fri, 12 May 2017 00:57:01 GMT

@Sonu Sahi

thanks for the response. The approach mentioned above does not give the ideal solution for my requirement.

Re: Parsing flow file or attribute in Nifi

MattWho — Fri, 12 May 2017 01:32:49 GMT

@Anil Reddy

Your new example does not use a "&" between all your fields. Is that a typo? I see "&" between first two fields and "," after that.

Re: Parsing flow file or attribute in Nifi

MattWho — Fri, 12 May 2017 01:44:31 GMT

@Anil Reddy

Are the four fields you are trying to extract the values from consistently:

"firstname"

"lastname"

"email"

"address"

Re: Parsing flow file or attribute in Nifi

anilcseiitm2006 — Fri, 12 May 2017 01:46:26 GMT

sorry, its a typo mistake.

firstname=testfirstname&lastname=testlastname&email=testemail@email.com&address=test address.

Re: Parsing flow file or attribute in Nifi

anilcseiitm2006 — Fri, 12 May 2017 01:49:44 GMT

yes, i want to extract values of firstname, lastname, email, address

we can assume like below.

If the flow files content is as

&firstname=testfirstname&lastname=testlastname&email=testemail@email.com&address=test address&

I want to extract XXXX in regex &firstname=XXXXXXXx&.* as value for firstname field

extract XXXXXx in regex .*&lastname=XXXXX&.* as value for lastname etc.

Re: Parsing flow file or attribute in Nifi

anilcseiitm2006 — Fri, 12 May 2017 01:50:32 GMT

Just for tagging!

@Matt Clarke

Re: Parsing flow file or attribute in Nifi

anilcseiitm2006 — Fri, 12 May 2017 02:06:52 GMT

@Matt Clarke

the structure or the fields order is not consistent.

I would like to put it this way, there are lot number of fields separated by '&' in the flow file and I just want to extract specific fields from the flow file content like

&testfield=testfieldvalue&testafield=testafieldvalue&... m number of fields...&firstname=firstnamevalue&... n number of fields ... &lastname=lastnamevalue& .... x number of fields ... &email=testemail@email.com& .... y number of fields .... &address=test address&...z number of fields .... &

Hope this gives better picture!

Re: Parsing flow file or attribute in Nifi

anilcseiitm2006 — Fri, 12 May 2017 02:33:09 GMT

@Matt Clarke

I got the answer. I can do something as below to extract the specific field value. For example for fisrtname, I do as below in ExtractText processor

.*&firstname=(.*?)&.*

but I see the copies of the attributes like firstname, firstname.0, firstname.1 etc. How can I mitigate the creation of rest of attributes and generate only "firstname".

Re: Parsing flow file or attribute in Nifi

MattWho — Sun, 18 Aug 2019 01:17:49 GMT

@Anil Reddy

You can get rid of capture group 0 by changing the "Include Capture Group 0" property to false.

Capture groups are defined using (). what falls between them is a capture group. You can create a single java expression that has multiple capture groups. So the ExtractText processor creates a attribute for each of these capture groups. Then the complete result of the regular expression is assigned to the property name you created.

In the above i have created four different java regular expressions. Each runs against the content of the incoming FlowFile. Each Regex has one capture group (.*?). As result you will have address.1= to that one defined capture group. NiFi also wants to create the attribute as you defined with just "address". so the <propertyname>.num attributes are created because of the use of capture groups in your regex.

For example, let assume the firstname property had the following regex:

firstname=(.*?)[&]{1}lastname=(.*?)[&]{1}

Here you see i have two capture groups in one regular expression.

This would translate in to two new attributes on your flowfile:

Thanks,

Matt

Re: Parsing flow file or attribute in Nifi

anilcseiitm2006 — Fri, 12 May 2017 03:27:11 GMT

Matt Clarke

thanks for the latest response. Its informative and gives more understanding about regular expressions!

Re: Parsing flow file or attribute in Nifi

MattWho — Sun, 18 Aug 2019 01:17:35 GMT

@Anil Reddy

Since you don't know where in the content these four fields will fall, I would suggest using the following:

The other regex rules I provided expect address as the last field.

these will work no matter where in the content these patterns are found.

The .*&firstname=(.*?)&.* will not work in all cases since it expect you to have 1 or more characters followed by "&filename=" followed by a many characters up to the last found "&" followed by 1 or more trailing characters. This expression will not work should firstname be the first or last field in the content.

Thank,

Matt If you found my answers helpful to your question, please mark it as "accepted"

Re: Parsing flow file or attribute in Nifi

alvinuw — Wed, 07 Jun 2017 03:36:43 GMT

Hi @Matt Clarke,

Does this one address=([a-zA-Z0-9 ,]+) mean NiFi can find the column named "address"?

I tried it, but no luck. Would you please upload an example?

Thanks.

Re: Parsing flow file or attribute in Nifi

MattWho — Wed, 07 Jun 2017 04:02:52 GMT

@Alvin Jin

My suggestion would be use something like:

https://regex101.com/

You can enter your regex and sample test you want to run it against.

Matt