Creating log files in nifi based on processor group

srilakshmi — Tue, 17 Jan 2023 07:14:33 GMT

Hi,

I wanted to create log files in nifi based on the processor group, could someone help me on the configuration that needs to be done in logback.xml file?

Like currently i have 3 different processor groups and i wanted to create 3 different log files for each of them.

Re: Creating log files in nifi based on processor group

MattWho — Tue, 17 Jan 2023 21:23:17 GMT

@srilakshmi
Logging does not happen at the process group level. Processors logging is based on the processor class. So there is nothing in the log output produced by a processor within a process group that is going to tell you in which process group that particular processor belongs.

That being said, you may be able to prefix every processor's name within the same Process group with some string that identifies the process group. This processor name would generally be included in the the log output produced by the processor. Then you may be able to use logback filters (have not tried this myself) to filter log output based on these unique strings.
https://logback.qos.ch/manual/filters.html

NiFi bulletins (bulletins are log output to the NiFi UI and have a rolling 5 minute life in the UI) however do include details about the parent Process Group in which the component generating the bulletin resides. You could build a dataflow in yoru NiFi to handle bulletin notification through the use of the SiteToSiteBulletinReportingTask which is used to send bulletin to a destination remote import port on a target NiFi. A dataflow on the target NiFi could be built to parse the received bulletin records by the bulletinGroupName json path property so that all records from same PG are kept together. These 'like' records could then be written out to local filesystem based on the PG name, remote system, used to send email notifications, etc...

Example of what a Bulletin sent using the SiteToSiteBulletinReportingTask looks like:

{
  "objectId" : "541dbd22-aa4b-4a1a-ad58-5d9a0b730e42",
  "platform" : "nifi",
  "bulletinId" : 2200,
  "bulletinCategory" : "Log Message",
  "bulletinGroupId" : "7e7ad459-0185-1000-ffff-ffff9e0b1503",
  "bulletinGroupName" : "PG2-Bulletin",
  "bulletinGroupPath" : "NiFi Flow / Matt's PG / PG2-Bulletin",
  "bulletinLevel" : "DEBUG",
  "bulletinMessage" : "UpdateAttribute[id=8c5b3806-9c3a-155b-ba15-260075ce9a6f] Updated attributes for StandardFlowFileRecord[uuid=1b0cb23a-75d8-4493-ba82-c6ea5c7d1ce3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1672661850924-5, container=default, section=5], offset=969194, length=1024],offset=0,name=bulletin-${nextInt()).txt,size=1024]; transferring to 'success'",
  "bulletinNodeId" : "e75bf99f-095c-4672-be53-bb5510b3eb5c",
  "bulletinSourceId" : "8c5b3806-9c3a-155b-ba15-260075ce9a6f",
  "bulletinSourceName" : "PG1-UpdateAttribute",
  "bulletinSourceType" : "PROCESSOR",
  "bulletinTimestamp" : "2023-01-04T20:38:27.776Z"
}

In the above produced bulletin json you see the BulletinGroupName and the BulletinMessage (the actual log output).

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

question Creating log files in nifi based on processor group in Support Questions

Creating log files in nifi based on processor group

Re: Creating log files in nifi based on processor group