What is best way to read nifi logs and put in MongoDb ?

[#1]

Hello Experts,

I have a requirement where nifi flows imp log information should be written to MongDB, so that another application can read mongodb and display those interesting logs info on web UI.

I am looking for best way to do this.

One way I am thinking of is tail nifi log (using tail file processor) and filter for imp log and put into MongoDb using native processor.

Please suggest if any other better approaches,

thanks in advance.

Thanks

Mahendra

[#2]

Your approach sounds perfectly reasonable if you only plan to use native nifi tools. I didn't fully understand what kind of filtering you meant, but simply reading the logs with the TailFile processor and sending them to mongo with the PutMongo processor sounds like it would work for your use-case.

[#3]

@Green_  Filtering means I may need to ignore some INFO level logs and consider only DEBUG or ERROR level logs... and I dont have to be using native nifi processors only.. but right way of doing is importtant.

Please suggest if you see an other way

[#4 · Accepted solution]

@hegdemahendra 

Filtering like that in nifi could work though it might be a bit resource intensive depending on the logs. 

Might be worth checking out a different tool specifically designed for handling logs. I've written logs to elasticsearch using logstash in the past,  perhaps it could also work for writing to a mongodb. 

[#5]

@Green_  - Can I have a custom appender configured in logback.xml to collect logs and put that in MongoDB ? would that be possible with nifi logs ?

[#6]

@hegdemahendra 

I have found this article by Pierre V. Where he goes into deeper detail about the logback.xml file. He mentions something that might be relevant to what you're looking for - the following two passages are what caught my eye:

```

"We can also define new appenders in the log configuration file and change it according to our needs. In particular, we could be interested by the SMTP Appender that can send logs via emails based on quite a large set of conditions. Full documentation here."

"Obviously  you can also configure this configuration file so that NiFi log files integrate with your existing systems. An idea could be to configure a Syslog appender to also redirect the logs to an external system."

```

I myself have never done something like this, but it sounds like a step in the right direction for directly writing logs from nifi to mongo.