question Re: How to filter logs according to terms using Apache Nifi? in Archives of Support Questions (Read Only)

How to filter logs according to terms using Apache Nifi?

xavwebmaster — Fri, 13 Oct 2017 18:07:41 GMT

Hello everyone,

I'm receiving logs as input, these I separate using the processor splittext in line by line.

I am interested in being able to divert these according to a certain field to different destinations.

I am not sure how to generate these attributes or conditions in order to derive the log lines using the routeonattribute processor, or routeontext depending on what you recommend.

Example of entries:

1467, monday, texas, allow, smtp, mail,etc

1468, monday, texas, deny, smtp, mail,etc

Expected output:

To Putkafka (If the line of log contains the term' allow')

To Putfile (If the line of og contains the term' deny')

Thank you very much

Re: How to filter logs according to terms using Apache Nifi?

ahadjidj — Fri, 13 Oct 2017 18:13:59 GMT

@xav webmaster

You need to generate flow file attributes based on the content of your data. These attrbutes will be used for routing in RouteOnAttribute. To construct these attributes you can use UpdateAttribute/ExtractText with Expression Langage.

If you just want to check if the line contains Allow or Deny you can use contains : https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#contains

If you want to verify a particular field then you need to parse the line. There are several methods to do it. For instance, you can use the function getDelimitedField since it's a CSV. Look also to record based processor. They have a ton of optimisation that can help you.

https://community.hortonworks.com/articles/102183/record-based-processors-in-apache-nifi-12.html

https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi

Re: How to filter logs according to terms using Apache Nifi?

xavwebmaster — Fri, 13 Oct 2017 18:29:59 GMT

Hi, thanks for the help. Then I understand that the scheme would be in: listensyslog > splittext > updatettribute > routeonatribute > putfile or putkafka

Re: How to filter logs according to terms using Apache Nifi?

ahadjidj — Sun, 18 Aug 2019 03:49:37 GMT

@xav webmaster

Can you test RouteOnContent with the property configured as follows :

This routes flow files based on the fact that a line contains "Allow" or not. You will have two relations : Allow and unmatched

Re: How to filter logs according to terms using Apache Nifi?

xavwebmaster — Fri, 13 Oct 2017 19:45:51 GMT

It worked perfectly. Thanks!