Nifi MergeRecord - can you merge on 2 different attributes.

Griggsy — Tue, 21 Apr 2026 11:23:32 GMT

Good afternoon

I am trying to make improvements to the way we make our Nifi flows by implementing Record processing.

Historically we:

split text > get timestamps using regex > merge on 'corellation_id' (attribute from timestamp: format- yyyy-MM-dd-HH) > extract sourcetype using regex (flow splits at this point for each sourcetype) > merge again on 'correlation_id' > out to HDFS/Splunk etc

flow structured like this:

Old Split/Merge Flow

(each spine is a different sourcetype)

I recently watched this series of videos

https://www.youtube.com/channel/UCcNsK56MZRXzPbRfe8sODWw

I was particularly interested in the first one where Record Processing is used in place of split/merge/regex etc.

As a result I have started to re-design the flow above, the logs come into us in CEF format from a syslogListener (TCP).

My progress so far in processors:

GenerateFlowfile - using some copied data from original flow.

SplitText - to split text/plain content into single logs (not figured out a way to get around splitting)

ParseCEF - parses CEF into JSON format

JoltTransformJSON - 'default' operation - to add some info to the JSON 'extension' content (feedname etc.)

JoltTransformJSON - ''shift' operation - to strip out unwanted lines, just keeping lines I later need as attributes + 'raw_content' line (I haven't figured out how to achieve these two steps using one Jolt processor? is there a way?)

EvaluateJsonPath - pulls the above attributes from the content.

MergeRecord - Currently merging on 'sourcetype'.....

That's as far as I have got, the flow is using a lot less processors and achieving (nearly) the same results. I have also compared the flow file linage duration, my new flow is around 5secs and the old style takes around 35secs (although could be tweaked).

WIP Process Record Flow

My question is, is there a way to merge on two attributes? 'sourcetype' and 'correlation_id' in my case I don't want different sourcetypes ("deviceVendor" in JSON format) merged together in HDFS and I also want to merge/group by timestamps (correlation_id) in one hour bins.

I want to try to get away from having 11 different spines for each different sourcetype, is the mergeRecord clever enough to group by common content and correlate from my time (correlation_id) attribute? or is this asking too much?

Hope this makes sense, any assistance would be greatly appreciated.

Re: Nifi MergeRecord - can you merge on 2 different attributes.

TimothySpann — Fri, 11 Dec 2020 15:30:04 GMT

You can make a new attribute with UpdateAttribute that merges those two together

question Nifi MergeRecord - can you merge on 2 different attributes. in Support Questions

Nifi MergeRecord - can you merge on 2 different attributes.

Re: Nifi MergeRecord - can you merge on 2 different attributes.