Nifi Processor group level logging-Issue

sarithe — Wed, 04 Jan 2023 17:14:41 GMT

Hi Team,

We are trying to enable logging at each processor group level. But we are unable to do it. From nifi-app.log we are getting as a general but we need log for each processor in a processor group level. We cannot use bulletin as well as bulletin will stay only for 5 min interval. Can anyone can help achieving it.

Re: Nifi Processor group level logging-Issue

MattWho — Wed, 04 Jan 2023 21:23:55 GMT

@sarithe NiFi component processors are part of pluggable nar in NiFi. They are separate from the core NiFi code. Processors are designed to log their output base on their component processor class. It then becomes the responsibility of the logback to route those log messages to the appropriate appender. There is nothing in the log output produced by a component processor that will inherently identify which parent process group it resides within. But if you were to use a consistent processor naming structure in each of your Process Groups (PG), you may be able to setup some creative filtering in logback based on that naming structure.

Bulletins however do include details about the parent Process Group in which the component generating the bulletin resides. You could build a dataflow in yoru NiFi to handle bulletin notification through the use of the SiteToSiteBulletinReportingTask which is used to send bulletin to a destination remote import port on a target NiFi. A dataflow on the target NiFi could be built to parse the received bulletin records by the bulletinGroupName json path property so that all records from same PG are kept together. These 'like' records could then be written out to local filesystem, remote system, used to send email notifications, etc...

Example of what a Bulletin sent using the SiteToSiteBulletinReportingTask looks like:

{ "objectId" : "541dbd22-aa4b-4a1a-ad58-5d9a0b730e42", "platform" : "nifi", "bulletinId" : 2200, "bulletinCategory" : "Log Message", "bulletinGroupId" : "7e7ad459-0185-1000-ffff-ffff9e0b1503", "bulletinGroupName" : "PG2-Bulletin", "bulletinGroupPath" : "NiFi Flow / Matt's PG / PG2-Bulletin", "bulletinLevel" : "DEBUG", "bulletinMessage" : "UpdateAttribute[id=8c5b3806-9c3a-155b-ba15-260075ce9a6f] Updated attributes for StandardFlowFileRecord[uuid=1b0cb23a-75d8-4493-ba82-c6ea5c7d1ce3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1672661850924-5, container=default, section=5], offset=969194, length=1024],offset=0,name=bulletin-${nextInt()).txt,size=1024]; transferring to 'success'", "bulletinNodeId" : "e75bf99f-095c-4672-be53-bb5510b3eb5c", "bulletinSourceId" : "8c5b3806-9c3a-155b-ba15-260075ce9a6f", "bulletinSourceName" : "PG1-UpdateAttribute", "bulletinSourceType" : "PROCESSOR", "bulletinTimestamp" : "2023-01-04T20:38:27.776Z" }

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Re: Nifi Processor group level logging-Issue

sarithe — Tue, 10 Jan 2023 13:17:20 GMT

Thanks Matt,I will test and Let you know

question Nifi Processor group level logging-Issue in Support Questions

Nifi Processor group level logging-Issue

Re: Nifi Processor group level logging-Issue

Re: Nifi Processor group level logging-Issue