Skip to main content
adhishankarit
adhishankarit
Gold ⭐️
December 14, 2020
Solved

How to track the Nifi procssors success or error status in shell script

Hi ,

I have to log Nifi processor success or errors status in shell script to monitor the flow . Please help with solution . I have tried using Updateattribute and log the message in a filesystem and could not complete it . Also i can not access nfi-app.log file into Edge node. Would like to do through REST API calls or some other solution . 

 

Thanks

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
December 24, 2020

@adhishankarit 


There is nothing you can pull form the NiFi Rest-API that is going to tell you about successful outcomes form processor execution on a FlowFile.  Depending on data volumes, this also sounds like a resource expensive endeavor.

That being said, NiFi does have a Site-To-SIte (S2S) Bulletin reporting task. When a processor throws an error it will produce a bulletin and this reporting task can capture those bulletins and send them via NiFi's S2S protocol to another NiFi instance directly into a dataflow were you can handle via dataflow design however you like.

Only way you can get INFO level logs in to bulletins is by setting the bulletin level to INFO on all you processors. This only works if you have also configured your NiFI logback.xml so that all NiFi components log at the INFO level as well.  Downsides to this:
1. Every processor would display the red bulletin square in upper right corner of processor, This makes using this to find components that are having issues difficult
2. This results in a lot of INFO level logging to the nifi-app.log.

You mention edge nodes.  You could setup a tailFile processor that tails the nifi-app.log and then send via a dataflow that log data via FlowFiles to some monitor NiFi cluster where another dataflows parses those records via a partitionRecord processor by log_level and then routes based on that log_level for additional handling/notification processing. Downside here:
1. Since you what to track success, you still need INFO level logging enabled for all components.  This means even this log collection flow is producing log output.  So large logs and logs being written to even when actual data being processed in other flows is not happening.

NiFi does have a master bulletin board which you could hit via the rest-api, but this does not get you past the massive logging you may be producing to monitor success.
https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

Hope this gives you some ideas,

Matt

View in thread

5 replies

justenji
justenji
Gold ⭐️⭐️
December 14, 2020

@adhishankarit 

 

I don't know wether I get your requirement right.
You want to log EACH single NiFi-processer regardless wether it succeeds successful or not (like error, retry, etc.). Right?

 

Evaluation of data of nifi-app.log (haven't done this by now) or by REST-API (I suppose something in the area "Flow" https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) might be possible.

 

But why don't you log this data direct from the flow by writing to a database logtable after each processor? Concerns about performance?

Translate
adhishankarit
adhishankaritAuthor
Gold ⭐️
December 14, 2020

@justenji 

We can not use database in our platform to log the message from the flow  .Can we use NFS fileystem to log the messages in a configuration file where it indicates success or failure with reason .if so could you please provide some nifi template .

 

Thank you 

Translate
justenji
justenji
Gold ⭐️⭐️
December 15, 2020

@adhishankarit 

 

Sadly I have no experience doing such kind of logging in own files to a NFS.

 

But what about logging to nifi-app.log? I don't understand the problem you mentioned in your first post concerning this.
Using the LogAttribute-processor gives you some options (attributes, content) what and how to log there. With the option "Log prefix" you can evaluate data by this value later.
IMHO Logging all attributes would be advantage because not all processors deliver the same attributes after they ran.

 

Maybe you can do something with scripts like mentioned here:
https://community.cloudera.com/t5/Support-Questions/Is-possible-to-write-an-attribute-into-a-file-and-also-keep/td-p/184414

 

Sorry but im neither fit with scripting nor with this kind of logging so I'm not able to help you further. But maybe some other guys here will help you.

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
December 24, 2020

@adhishankarit 


There is nothing you can pull form the NiFi Rest-API that is going to tell you about successful outcomes form processor execution on a FlowFile.  Depending on data volumes, this also sounds like a resource expensive endeavor.

That being said, NiFi does have a Site-To-SIte (S2S) Bulletin reporting task. When a processor throws an error it will produce a bulletin and this reporting task can capture those bulletins and send them via NiFi's S2S protocol to another NiFi instance directly into a dataflow were you can handle via dataflow design however you like.

Only way you can get INFO level logs in to bulletins is by setting the bulletin level to INFO on all you processors. This only works if you have also configured your NiFI logback.xml so that all NiFi components log at the INFO level as well.  Downsides to this:
1. Every processor would display the red bulletin square in upper right corner of processor, This makes using this to find components that are having issues difficult
2. This results in a lot of INFO level logging to the nifi-app.log.

You mention edge nodes.  You could setup a tailFile processor that tails the nifi-app.log and then send via a dataflow that log data via FlowFiles to some monitor NiFi cluster where another dataflows parses those records via a partitionRecord processor by log_level and then routes based on that log_level for additional handling/notification processing. Downside here:
1. Since you what to track success, you still need INFO level logging enabled for all components.  This means even this log collection flow is producing log output.  So large logs and logs being written to even when actual data being processed in other flows is not happening.

NiFi does have a master bulletin board which you could hit via the rest-api, but this does not get you past the massive logging you may be producing to monitor success.
https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

Hope this gives you some ideas,

Matt

Translate
Terms & ConditionsAccessibility statement