Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to filter logs according to terms using Apache Nifi?

avatar
Contributor

Hello everyone,

I'm receiving logs as input, these I separate using the processor splittext in line by line.

I am interested in being able to divert these according to a certain field to different destinations.

I am not sure how to generate these attributes or conditions in order to derive the log lines using the routeonattribute processor, or routeontext depending on what you recommend.

Example of entries:

1467, monday, texas, allow, smtp, mail,etc

1468, monday, texas, deny, smtp, mail,etc

Expected output:

To Putkafka (If the line of log contains the term' allow')

To Putfile (If the line of og contains the term' deny')

Thank you very much

1 ACCEPTED SOLUTION

avatar
@xav webmaster

Can you test RouteOnContent with the property configured as follows :

40827-screen-shot-2017-10-13-at-20117-pm.png

This routes flow files based on the fact that a line contains "Allow" or not. You will have two relations : Allow and unmatched

View solution in original post

4 REPLIES 4

avatar

@xav webmaster

You need to generate flow file attributes based on the content of your data. These attrbutes will be used for routing in RouteOnAttribute. To construct these attributes you can use UpdateAttribute/ExtractText with Expression Langage.

If you just want to check if the line contains Allow or Deny you can use contains : https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#contains

If you want to verify a particular field then you need to parse the line. There are several methods to do it. For instance, you can use the function getDelimitedField since it's a CSV. Look also to record based processor. They have a ton of optimisation that can help you.

https://community.hortonworks.com/articles/102183/record-based-processors-in-apache-nifi-12.html

https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi

avatar
Contributor

Hi, thanks for the help. Then I understand that the scheme would be in: listensyslog > splittext > updatettribute > routeonatribute > putfile or putkafka

avatar
@xav webmaster

Can you test RouteOnContent with the property configured as follows :

40827-screen-shot-2017-10-13-at-20117-pm.png

This routes flow files based on the fact that a line contains "Allow" or not. You will have two relations : Allow and unmatched

avatar
Contributor
It worked perfectly. Thanks!