Nifi: How to get the error generated at bulletin board for a specific flowfile.

[#1]

I am trying to read the error from bulletin board using InvokeHttp processor for a specific flowfile. But i am not able to get the error which is generated for a flowfile as it gives either all errors or error belonging to a processGroup(using rest-api to get processGroup id at root canvas for remote url ). The error has sourceId (id of processor generating error) and group id (id of ProcessGroup where error generating processor is located).

How can we get the sourceId(highlighted below) and ProcessGroup id as attribute with flowfile?

Is there any other way to read only that error which is being generated on bulletin-board for specific flowfile ?

[#2]

@MahipalRathore 

Rather then trying use the invokeHTTP processor trying to fetch bulletins as they happen, I suggest you use the SiteToSiteBulletinReportingTask to send bulletins as they occur to a remote input port on your NiFi or another NiFi setup fro monitoring.   You can then build a dataflow to filter and route the received bulletins as you desire.

SiteToSiteBulletinReportingTask

The Site-to-Site Bulletin Reporting Task allows the user to publish Bulletin events using the Site To Site protocol. Note: only up to 5 bulletins are stored per component and up to 10 bulletins at controller level for a duration of up to 5 minutes. If this reporting task is not scheduled frequently enough some bulletins may not be sent.

Thanks,

Matt

[#3]

Hi Matt,

I am able to use SiteToSiteBulletinReportingTask to consume error from bulletin board. I can get the bulletinGroupId, bulletinGroupName, bulletinMessage, bulletinSourceId, bulletinSourceName, bulletinSourceType and bulletinTimestamp attributes. But i am not able to figure out the way to get all flowfile attributes or flowfile details for which the failure has occured. For example: I am ingesting a file to hdfs from local directory , it will have some attribute like filename, location, size etc. and when some failure occurs while placing on hdfs, I must get (filename, location, size ) attributes along with failure details.

Thanks,

Mahipal 

[#4]

@MahipalRathore 

The bulletin is not going to include anything more than what would be found in the nifi-app.log.  The Bulletin if produced as a result of some failure while processing a FlowFile, will have details about the FlowFiles assigned UUID and filename as well as it size relative to content claim in which the FlowFiles content can be found.  You could the use NiFi data provenance to obtain details on the FlowFile including all its FlowFile attributes.  Some bulletins are for exceptions that occur unrelated to a FlowFile directly and thus will contain no FlowFile info.  But if a FlowFile is routed to failure as a result of for example an exception thrown by the putHDFS processor, details on that FlowFile record should be included in the bulletin.

Note: If the bulletin is being produced by a processor that creates a FlowFile, no FlowFile will have been created, so there is no FlowFile created from which to get FlowFile details.

Hope this helps,

Matt