custom cisco syslog to cef format

[#1]

Hello,

We have custom syslog Cisco messages in the following format 

<117>2024-07-23T14:09:56Z XXXXXXXXX : %FTD-5-430003: EventPriority: Low, DeviceUUID: xxxxxxxxxxxxx, InstanceID: 2, FirstPacketSecond: 2024-07-23T14:09:56Z, ConnectionID: 32322, AccessControlRuleAction: Allow, SrcIP: A.B.C.D, DstIP: A.B.C.D, SrcPort: 42308, DstPort: 24224, Protocol: tcp, IngressInterface: XXX, EgressInterface: XXX, IngressZone: XXX, EgressZone: YYY, IngressVRF: Global, EgressVRF: Global, ACPolicy: AAA-BBB, 

We want to use NIFI to format to CEF ( common event format ). Any help which processors to use, please?

[#2 · Accepted solution]

Hi @cadrian90 ,

Im not aware of direct way to do that in Nifi. I know there are services\processor like CEFFeader and ParseCEF used to consume CEF format but not to write as CEF. the good news is that you can write your custom code to create service or new processor  to do that using Either Python or Java if you happen to know a way of doing using code.

[#3]

@cadrian90 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @SAMSAL @MattWho  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#4 · Accepted solution]

Hi @cadrian90 ,

Im not aware of direct way to do that in Nifi. I know there are services\processor like CEFFeader and ParseCEF used to consume CEF format but not to write as CEF. the good news is that you can write your custom code to create service or new processor  to do that using Either Python or Java if you happen to know a way of doing using code.

[#5 · Accepted solution]

@cadrian90 

I agree with @SAMSAL response.  Typically the ConvertRecord processor is what would be used here.  The processor support numerous record readers and numerous record writers.  The GrokReader is what would be commonly used to parse unstructured data like your Cisco syslog messages.  While the GrokReader has bulit in pattern file, you may fond yourself needing to define a custom pattern file for your specific data.  You might find this other community post helpful here:
https://community.cloudera.com/t5/Support-Questions/ExtractGrok-processor-Writing-Regex-to-parse-Cis...

Beyond above, this is where it becomes challenging since Apache NiFi only has a CEFReader and no CEFRecordSetWriter (perhaps you can raise an Apache Jira asking for this new reader and someone in the Apache community may be able to help)

There does exist a ScriptedRecordSetWriter that if you know how to scripted out the CEF format, maybe you can use that.  I really would not be able to help there myself.
Maybe you can look into the CSVRecordSetWriter to see if selecting a custom format would facilitate an output like CEF.  Again not something I have tried myself.

Hope this helps you with your use case journey.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#6]

@cadrian90 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  Thanks.