How to filter logs according to terms using Apache Nifi?

[#1]

Hello everyone,

I'm receiving logs as input, these I separate using the processor splittext in line by line.

I am interested in being able to divert these according to a certain field to different destinations.

I am not sure how to generate these attributes or conditions in order to derive the log lines using the routeonattribute processor, or routeontext depending on what you recommend.

Example of entries:

1467, monday, texas, allow, smtp, mail,etc

1468, monday, texas, deny, smtp, mail,etc

Expected output:

To Putkafka (If the line of log contains the term' allow')

To Putfile (If the line of og contains the term' deny')

Thank you very much

[#2]

@xav webmaster

You need to generate flow file attributes based on the content of your data. These attrbutes will be used for routing in RouteOnAttribute. To construct these attributes you can use UpdateAttribute/ExtractText with Expression Langage.

If you just want to check if the line contains Allow or Deny you can use contains : https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#contains

If you want to verify a particular field then you need to parse the line. There are several methods to do it. For instance, you can use the function getDelimitedField since it's a CSV. Look also to record based processor. They have a ton of optimisation that can help you.

https://community.hortonworks.com/articles/102183/record-based-processors-in-apache-nifi-12.html

https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi

[#3]

Hi, thanks for the help. Then I understand that the scheme would be in: listensyslog > splittext > updatettribute > routeonatribute > putfile or putkafka

[#4 · Accepted solution]

@xav webmaster

Can you test RouteOnContent with the property configured as follows :

This routes flow files based on the fact that a line contains "Allow" or not. You will have two relations : Allow and unmatched

[#5]

It worked perfectly. Thanks!