Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi MergeRecord - can you merge on 2 different attributes.

avatar
Contributor

Good afternoon

 

I am trying to make improvements to the way we make our Nifi flows by implementing Record processing.

 

Historically we:

split text > get timestamps using regex > merge on 'corellation_id' (attribute from timestamp: format- yyyy-MM-dd-HH) > extract sourcetype using regex (flow splits at this point for each sourcetype) > merge again on 'correlation_id' > out to HDFS/Splunk etc

 

flow structured like this:

Old  Split/Merge FlowOld Split/Merge Flow

(each spine is a different sourcetype)

 

I recently watched this series of videos

https://www.youtube.com/channel/UCcNsK56MZRXzPbRfe8sODWw

I was particularly interested in the first one where Record Processing is used in place of split/merge/regex etc. 

 

As a result I have started to re-design the flow above, the logs come into us in CEF format from a syslogListener (TCP).

 

My progress so far in processors:

 

GenerateFlowfile - using some copied data from original flow.

SplitText - to split text/plain content into single logs (not figured out a way to get around splitting)

ParseCEF - parses CEF into JSON format

JoltTransformJSON - 'default' operation - to add some info to the JSON 'extension' content (feedname etc.) 

JoltTransformJSON - ''shift' operation - to strip out unwanted lines, just keeping lines I later need as attributes  + 'raw_content' line (I haven't figured out how to achieve these two steps using one Jolt processor? is there a way?)

EvaluateJsonPath - pulls the above attributes from the content.

MergeRecord - Currently merging on 'sourcetype'.....

 

That's as far as I have got,  the flow is using a lot less processors and achieving (nearly) the same results. I have also compared the flow file linage duration, my new flow is around 5secs and the old style takes around 35secs (although could be tweaked).

 

WIP Process Record FlowWIP Process Record Flow

 

 

My question is, is there a way to merge on two attributes? 'sourcetype' and 'correlation_id' in my case I don't want different sourcetypes ("deviceVendor" in JSON format) merged together in HDFS and I also want to merge/group by timestamps (correlation_id) in one hour bins.

 

I want to try to get away from having 11 different spines for each different sourcetype, is the mergeRecord clever enough to group by common content and correlate from my time (correlation_id) attribute? or is this asking too much?

 

Hope this makes sense, any assistance would be greatly appreciated. 

1 ACCEPTED SOLUTION

avatar
Master Guru

You can make a new attribute with UpdateAttribute that merges those two together

View solution in original post

1 REPLY 1

avatar
Master Guru

You can make a new attribute with UpdateAttribute that merges those two together