NiFi - Problems merging flow file content and attributes with replace text processor

[#1]

Hi,

I have some problems merging the response of a API call with InvokeHTTP to the orginal flow file. I tried to put the response as an attribute to the original file and sent it to an replace text processor. I set the Evaluation Mode to Entire Text and tried either prepend and append replacement strategy.

In prepend mode the processor is just passing the attributes as new content and in append the processor is writing the original content to the flow files. Any ideas how to merge the both datasets?

[#2 · Accepted solution]

@Lukas Müller,

For your case we are having 2 json responses and trying to merge them as one, we need to do some kind of preparing the json messages before doing merging them as one.

1.You need to remove the Closed braces after sensordatavalues array you need to remove the last close braces } from the json response.

by using replace text processor with search value as (.*)}\Z and Replacement value as $1 Replacement strategy as Regex Replace.

Configs:-

Then extract the resultant content as attribute to the flow file.

2.Then trigger invoke http processor once you get the result.

Right now we are having a json response but we need to merge our extracted output to the response and make it as one json message

3.for json response that you got from above invoke http processor we need to remove the open braces { with Replace Text processor.

by using search value as ^[{](.*) and Replacement value as $1 Replacement strategy as Regex Replace.

Configs:-

so the response from the invoke http processor we have removed the first open braces then use

4. Replace Text processor to prepare your required json message now by using Prepend with the attributed that we have extracted with extract text processor.

Example:-

Extract text processor attribute value:- i'm keeping this attribute name as extracted_Attribute for testing

{  "id" : 590929682,  "sampling_rate" : null,  "timestamp" : "2017-12-19 16:00:58",  "location" : {  "id" : 2191,  "latitude" : "51.482",  "longitude" : "7.408",  "country" : "DE"  },  "sensor" : {  "id" : 4355,  "pin" : "1",  "sensor_type" : {  "id" : 14,  "name" : "SDS011",  "manufacturer" : "Nova Fitness"  }  },  "sensordatavalues" : [ {  "id" : 1292433314,  "value" : "90.53",  "value_type" : "P1"  } ]

like this way with out closing braces at the end

Then next replace text processor we are removing the first open braces then the content should be like this way

"coord" : {  "lon" : 8.92,  "lat" : 48.85  },  "weather" : [ {  "id" : 300,  "main" : "Drizzle",  "description" : "light intensity drizzle",  "icon" : "09d"  }, {  "id" : 701,  "main" : "Mist",  "description" : "mist",  "icon" : "50d"  } ],  "base" : "stations",  "main" : {  "temp" : 276.69,  "pressure" : 1033,  "humidity" : 93,  "temp_min" : 276.15,  "temp_max" : 277.15  },  "visibility" : 10000,  "wind" : {  "speed" : 1  },  "clouds" : {  "all" : 75  },  "dt" : 1513765200,  "sys" : {  "type" : 1,  "id" : 4891,  "message" : 0.01,  "country" : "DE",  "sunrise" : 1513754085,  "sunset" : 1513783780  },  "id" : 2812053,  "name" : "Weissach",  "cod" : 200}

Then the next replace text processor which is going to prepare the merged json message properties would be like

For testing purpose i did this but you change the prepend and appends as per your json responses.

This is the way we can do merge 2 json responses and make it as one valid json message by using NiFi processors.

[#3]

@Shu

Thanks for you detailed reply. That seems very helpful to my porpuse. How did you attech the resulting content as attribute to the flow file?

[#4]

@Lukas Müller,
Use Replace text processor with above Configs as i shown in last attached screenshot then the Output of the processor will give you merged json message.

[#5]

@Shu I don't quiet understand that part because you mentioned extracting the content with an extract text processor? How did you implement that?

[#6]

@Lukas Müller

i think in your flow you are having 2 json responses

1. before invoke http processor(until 4 processor in the screenshot below)
2. after invoke http processor(after 4 processor in the screenshot)

Flow:-

1. Replace Text //to remove last closed braces using replace text processor)
2. Extract text //extract the content of the flowfile and keep as attribute
3. invoke http 
4. Replace Text //To remove first Open braces { for the response of invoke http processor
5. Replace Text //merge the existing content of the flowfile with the extracted attribute(step2) 

Let me know if you are not able to understand ..!!

[#7]

@Shu

Could you show me the configuration off your Extract Content as Attribute Processor? I think thats the point where I'm struggling

[#8]

@Lukas Müller

If your content is in more than one new line then you need to change the following properties to True. by default these properties are false

Enable DOTALL Mode

false

true false

Indicates that the expression '.' should match any character, including a line terminator. Can also be specified via the embedded flag (?s).

Enable Multiline Mode

false

true false

Indicates that '^' and ' should match just after and just before a line terminator or end of sequence, instead of only the beginning or end of the entire input. Can also be specified via the embeded flag (?m).

2.You need to change the below property if your extract content is more than 1024 B(1 KB)

Maximum Capture Group Length

1024

Specifies the maximum number of characters a given capture group value can have. Any characters beyond the max will be truncated.

3. To extract whole content of the flowfile add new property by clicking on + sign at right corner

extract_Attribute

(.*) //matches everything

[#9]

@Shu that did the trick! Many thanks for your help and your patience as well.