Skip to main content
Madhav_VD
Madhav_VD
Silver ⭐️
July 17, 2023
Solved

How to save NIFI Error log to database table ?

How to save NIFI Error log to database table ? 

steven-matison
steven-matisonAnswer
Platinum ⭐️⭐️
July 17, 2023

@Madhav_VD This is a job that nifi can do itself.  You would just tail the log file w/ nifi TailFile. 

 

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.TailFile/

 

 

 Once you have the log file going into flowfiles in nifi, you can build your flow to filter lines you want to versus lines you do not want.  Then when ready use any flavor of database processor (ExecuteSQLRecord, ExecuteSQL, PutSQL, PutDatabase, etc) to translate the log line into a sql insert statement.

View in thread

5 replies

VidyaSargur
Community Manager
VidyaSargur
Community Manager
July 17, 2023

@Madhav_VD, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @cotopaul @SAMSAL @MattWho who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

 

Translate
Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
steven-matison
Platinum ⭐️⭐️
steven-matisonAnswer
Platinum ⭐️⭐️
July 17, 2023

@Madhav_VD This is a job that nifi can do itself.  You would just tail the log file w/ nifi TailFile. 

 

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.TailFile/

 

 

 Once you have the log file going into flowfiles in nifi, you can build your flow to filter lines you want to versus lines you do not want.  Then when ready use any flavor of database processor (ExecuteSQLRecord, ExecuteSQL, PutSQL, PutDatabase, etc) to translate the log line into a sql insert statement.

Translate
RabidRacoon
RabidRacoon
Silver ⭐️
November 14, 2023

We did this at our end and ended up re-cycling the provenance repository much faster than usual. The huge amount of data that an output of a tailfile generates can fill up both your content and provenance repositories.

Translate
SAMSAL
SAMSAL
Platinum ⭐️⭐️⭐️
July 17, 2023

Hi @Madhav_VD ,

In addition to what @steven-matison has mentioned which will work, there is another option where you can capture errors through setting up "SiteToSiteBulletinReportingTask" which will channel all errors to an input port where you can capture the error information in json format from all processors and all nodes in cluster and then store that error in sql or send notification email. This option might require additional steps to set up but its worth it specially if you have cluster. For more info on how to set up the "SiteToSiteBulletinReportingTask" please refer to : https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/

Hope that helps

 

 

Translate
VidyaSargur
Community Manager
VidyaSargur
Community Manager
July 21, 2023

@Madhav_VD, Have any of the replies helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

 

Translate
Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
Terms & ConditionsAccessibility statement