question Re: What is best way to read nifi logs and put in MongoDb ? in Support Questions

What is best way to read nifi logs and put in MongoDb ?

hegdemahendra — Wed, 04 Aug 2021 16:36:28 GMT

Hello Experts,

I have a requirement where nifi flows imp log information should be written to MongDB, so that another application can read mongodb and display those interesting logs info on web UI.

I am looking for best way to do this.

One way I am thinking of is tail nifi log (using tail file processor) and filter for imp log and put into MongoDb using native processor.

Please suggest if any other better approaches,

thanks in advance.

Thanks

Mahendra

Re: What is best way to read nifi logs and put in MongoDb ?

Green_ — Wed, 04 Aug 2021 19:44:05 GMT

Your approach sounds perfectly reasonable if you only plan to use native nifi tools. I didn't fully understand what kind of filtering you meant, but simply reading the logs with the TailFile processor and sending them to mongo with the PutMongo processor sounds like it would work for your use-case.

Re: What is best way to read nifi logs and put in MongoDb ?

hegdemahendra — Thu, 05 Aug 2021 05:43:08 GMT

@Green_ Filtering means I may need to ignore some INFO level logs and consider only DEBUG or ERROR level logs... and I dont have to be using native nifi processors only.. but right way of doing is importtant.

Please suggest if you see an other way

Re: What is best way to read nifi logs and put in MongoDb ?

Green_ — Thu, 05 Aug 2021 06:08:48 GMT

@hegdemahendra

Filtering like that in nifi could work though it might be a bit resource intensive depending on the logs.

Might be worth checking out a different tool specifically designed for handling logs. I've written logs to elasticsearch using logstash in the past, perhaps it could also work for writing to a mongodb.

Re: What is best way to read nifi logs and put in MongoDb ?

hegdemahendra — Mon, 09 Aug 2021 06:51:48 GMT

@Green_ - Can I have a custom appender configured in logback.xml to collect logs and put that in MongoDB ? would that be possible with nifi logs ?

Re: What is best way to read nifi logs and put in MongoDb ?

Green_ — Tue, 10 Aug 2021 21:12:42 GMT

@hegdemahendra

I have found this article by Pierre V. Where he goes into deeper detail about the logback.xml file. He mentions something that might be relevant to what you're looking for - the following two passages are what caught my eye:

```

"We can also define new appenders in the log configuration file and change it according to our needs. In particular, we could be interested by the SMTP Appender that can send logs via emails based on quite a large set of conditions. Full documentation here."

"Obviously you can also configure this configuration file so that NiFi log files integrate with your existing systems. An idea could be to configure a Syslog appender to also redirect the logs to an external system."

```

I myself have never done something like this, but it sounds like a step in the right direction for directly writing logs from nifi to mongo.