Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi - Problems merging flow file content and attributes with replace text processor

Solved Go to solution
Highlighted

NiFi - Problems merging flow file content and attributes with replace text processor

New Contributor

Hi,

I have some problems merging the response of a API call with InvokeHTTP to the orginal flow file. I tried to put the response as an attribute to the original file and sent it to an replace text processor. I set the Evaluation Mode to Entire Text and tried either prepend and append replacement strategy.

In prepend mode the processor is just passing the attributes as new content and in append the processor is writing the original content to the flow files. Any ideas how to merge the both datasets?

45497-2017-12-20-112826-1920x1103-scrot.png

45498-2017-12-20-112822-1920x1103-scrot.png

45496-2017-12-20-112839-1920x1103-scrot.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: NiFi - Problems merging flow file content and attributes with replace text processor

Super Guru

@Lukas Müller,

For your case we are having 2 json responses and trying to merge them as one, we need to do some kind of preparing the json messages before doing merging them as one.

1.You need to remove the Closed braces after sensordatavalues array you need to remove the last close braces } from the json response.

by using replace text processor with search value as (.*)}\Z and Replacement value as $1 Replacement strategy as Regex Replace.

Configs:-

45511-replacetext1.png

Then extract the resultant content as attribute to the flow file.

2.Then trigger invoke http processor once you get the result.

Right now we are having a json response but we need to merge our extracted output to the response and make it as one json message

3.for json response that you got from above invoke http processor we need to remove the open braces { with Replace Text processor.

by using search value as ^[{](.*) and Replacement value as $1 Replacement strategy as Regex Replace.

Configs:-

45512-replacetext2.png

so the response from the invoke http processor we have removed the first open braces then use

4. Replace Text processor to prepare your required json message now by using Prepend with the attributed that we have extracted with extract text processor.

Example:-

Extract text processor attribute value:- i'm keeping this attribute name as extracted_Attribute for testing

{  "id" : 590929682,  "sampling_rate" : null,  "timestamp" : "2017-12-19 16:00:58",  "location" : {  "id" : 2191,  "latitude" : "51.482",  "longitude" : "7.408",  "country" : "DE"  },  "sensor" : {  "id" : 4355,  "pin" : "1",  "sensor_type" : {  "id" : 14,  "name" : "SDS011",  "manufacturer" : "Nova Fitness"  }  },  "sensordatavalues" : [ {  "id" : 1292433314,  "value" : "90.53",  "value_type" : "P1"  } ]

like this way with out closing braces at the end

Then next replace text processor we are removing the first open braces then the content should be like this way

"coord" : {  "lon" : 8.92,  "lat" : 48.85  },  "weather" : [ {  "id" : 300,  "main" : "Drizzle",  "description" : "light intensity drizzle",  "icon" : "09d"  }, {  "id" : 701,  "main" : "Mist",  "description" : "mist",  "icon" : "50d"  } ],  "base" : "stations",  "main" : {  "temp" : 276.69,  "pressure" : 1033,  "humidity" : 93,  "temp_min" : 276.15,  "temp_max" : 277.15  },  "visibility" : 10000,  "wind" : {  "speed" : 1  },  "clouds" : {  "all" : 75  },  "dt" : 1513765200,  "sys" : {  "type" : 1,  "id" : 4891,  "message" : 0.01,  "country" : "DE",  "sunrise" : 1513754085,  "sunset" : 1513783780  },  "id" : 2812053,  "name" : "Weissach",  "cod" : 200}

Then the next replace text processor which is going to prepare the merged json message properties would be like

45513-replacetext.png

For testing purpose i did this but you change the prepend and appends as per your json responses.

This is the way we can do merge 2 json responses and make it as one valid json message by using NiFi processors.

8 REPLIES 8

Re: NiFi - Problems merging flow file content and attributes with replace text processor

Super Guru

@Lukas Müller,

For your case we are having 2 json responses and trying to merge them as one, we need to do some kind of preparing the json messages before doing merging them as one.

1.You need to remove the Closed braces after sensordatavalues array you need to remove the last close braces } from the json response.

by using replace text processor with search value as (.*)}\Z and Replacement value as $1 Replacement strategy as Regex Replace.

Configs:-

45511-replacetext1.png

Then extract the resultant content as attribute to the flow file.

2.Then trigger invoke http processor once you get the result.

Right now we are having a json response but we need to merge our extracted output to the response and make it as one json message

3.for json response that you got from above invoke http processor we need to remove the open braces { with Replace Text processor.

by using search value as ^[{](.*) and Replacement value as $1 Replacement strategy as Regex Replace.

Configs:-

45512-replacetext2.png

so the response from the invoke http processor we have removed the first open braces then use

4. Replace Text processor to prepare your required json message now by using Prepend with the attributed that we have extracted with extract text processor.

Example:-

Extract text processor attribute value:- i'm keeping this attribute name as extracted_Attribute for testing

{  "id" : 590929682,  "sampling_rate" : null,  "timestamp" : "2017-12-19 16:00:58",  "location" : {  "id" : 2191,  "latitude" : "51.482",  "longitude" : "7.408",  "country" : "DE"  },  "sensor" : {  "id" : 4355,  "pin" : "1",  "sensor_type" : {  "id" : 14,  "name" : "SDS011",  "manufacturer" : "Nova Fitness"  }  },  "sensordatavalues" : [ {  "id" : 1292433314,  "value" : "90.53",  "value_type" : "P1"  } ]

like this way with out closing braces at the end

Then next replace text processor we are removing the first open braces then the content should be like this way

"coord" : {  "lon" : 8.92,  "lat" : 48.85  },  "weather" : [ {  "id" : 300,  "main" : "Drizzle",  "description" : "light intensity drizzle",  "icon" : "09d"  }, {  "id" : 701,  "main" : "Mist",  "description" : "mist",  "icon" : "50d"  } ],  "base" : "stations",  "main" : {  "temp" : 276.69,  "pressure" : 1033,  "humidity" : 93,  "temp_min" : 276.15,  "temp_max" : 277.15  },  "visibility" : 10000,  "wind" : {  "speed" : 1  },  "clouds" : {  "all" : 75  },  "dt" : 1513765200,  "sys" : {  "type" : 1,  "id" : 4891,  "message" : 0.01,  "country" : "DE",  "sunrise" : 1513754085,  "sunset" : 1513783780  },  "id" : 2812053,  "name" : "Weissach",  "cod" : 200}

Then the next replace text processor which is going to prepare the merged json message properties would be like

45513-replacetext.png

For testing purpose i did this but you change the prepend and appends as per your json responses.

This is the way we can do merge 2 json responses and make it as one valid json message by using NiFi processors.

Re: NiFi - Problems merging flow file content and attributes with replace text processor

New Contributor

@Shu

Thanks for you detailed reply. That seems very helpful to my porpuse. How did you attech the resulting content as attribute to the flow file?

Re: NiFi - Problems merging flow file content and attributes with replace text processor

Super Guru

@Lukas Müller,
Use Replace text processor with above Configs as i shown in last attached screenshot then the Output of the processor will give you merged json message.

Re: NiFi - Problems merging flow file content and attributes with replace text processor

New Contributor

@Shu I don't quiet understand that part because you mentioned extracting the content with an extract text processor? How did you implement that?

Re: NiFi - Problems merging flow file content and attributes with replace text processor

Super Guru
@Lukas Müller

i think in your flow you are having 2 json responses

1. before invoke http processor(until 4 processor in the screenshot below)
2. after invoke http processor(after 4 processor in the screenshot)

Flow:-

1. Replace Text //to remove last closed braces using replace text processor)
2. Extract text //extract the content of the flowfile and keep as attribute
3. invoke http
4. Replace Text //To remove first Open braces { for the response of invoke http processor
5. Replace Text //merge the existing content of the flowfile with the extracted attribute(step2)

45525-flow.png

Let me know if you are not able to understand ..!!

Re: NiFi - Problems merging flow file content and attributes with replace text processor

New Contributor

@Shu

Could you show me the configuration off your Extract Content as Attribute Processor? I think thats the point where I'm struggling

Re: NiFi - Problems merging flow file content and attributes with replace text processor

Super Guru

@Lukas Müller

45527-extracttext.png

If your content is in more than one new line then you need to change the following properties to True. by default these properties are false

Enable DOTALL Modefalse
  • true
  • false
Indicates that the expression '.' should match any character, including a line terminator. Can also be specified via the embedded flag (?s).
Enable Multiline Modefalse
  • true
  • false
Indicates that '^' and ' should match just after and just before a line terminator or end of sequence, instead of only the beginning or end of the entire input. Can also be specified via the embeded flag (?m).

2.You need to change the below property if your extract content is more than 1024 B(1 KB)

Maximum Capture Group Length1024Specifies the maximum number of characters a given capture group value can have. Any characters beyond the max will be truncated.

3. To extract whole content of the flowfile add new property by clicking on + sign at right corner

extract_Attribute

(.*) //matches everything

Re: NiFi - Problems merging flow file content and attributes with replace text processor

New Contributor

@Shu that did the trick! Many thanks for your help and your patience as well.

Don't have an account?
Coming from Hortonworks? Activate your account here