Site to Site Status Reporting Task Error Notification Issue and JSON Output Format

[#1]

Please, I have a couple of questions. To put it in context, I need to capture only the flows that are running with errors, and I'm wondering how or what the recommended configuration would be for configuring the Site-to-Site Status Reporting Task controller for a 3-node cluster. As it is, it generates a lot of error messages and fills up my disk. Therefore, I want to suppress those messages with the correct configuration.

node-1:8080SiteToSiteStatusReportingTask[id=743817f2-0199-1000-0000-000026cab202] EndpointConnectionPool[Cluster URL=[http://172.16.7.xxx:8080/nifi-api]] failed to communicate with Peer[url=nifi://node-3:2888,CLOSED] due to java.io.EOFException

node-2:8080SiteToSiteStatusReportingTask[id=743817f2-0199-1000-0000-000026cab202] EndpointConnectionPool[Cluster URL=[http://172.16.7.xxx:8080/nifi-api]] failed to communicate with Peer[url=nifi://node-3:2888,CLOSED] due to java.io.EOFException

All NodesSiteToSiteStatusReportingTask[id=743817f2-0199-1000-0000-000026cab202] Error running task SiteToSiteStatusReportingTask[id=743817f2-0199-1000-0000-000026cab202] due to org.apache.nifi.processor.exception.ProcessException: Failed to send Status Records to destination due to IOException:null

Finally, I received a couple of captured records, but they don't arrive in the same JSON format. First, they arrived with these keys, whose information they contained is perfect, and I created the table where I captured them.

JSON 1
objectid,
bulletincategory,
bulletingroupname,
bulletingrouppath,
bulletinlevel,
bulletinmessage,
bulletinsourcename,
bulletinsourcetype,
bulletintimestamp,
date_update

But now I've changed it. They arrive in this other JSON format and they don't fit in my table:

json 2
statusId
timestampMillis
timestamp
actorHostname
componentType
componentName
parentId
parentName
parentPath
platform
applications
componentId
processorType
averageLineageDurationMS
bytesRead
bytesWritten
bytesReceived
bytesSent
flowFilesRemoved
flowFilesReceived
flowFilesSent
inputCount
inputBytes
outputCount
outputBytes
activeThreadCount
terminatedThreadCount
invocations
processingNanos
runStatus
executionNode
counters

Please any help with this I would appreciate your support.

Best regards to all.

Nfi.properties

Flow

Errors:

Setting Controller S2S:

[#2 · Accepted solution]

@Bern 

The two outputs you shared are form two different Site-To-Site Reporting tasks.

The first you shared is produced by the SiteToSiteBulletinReportingTask.
Additional Details...

The second you shared is produced by the SiteToSiteStatusReportingTask.
Its fields will vary based upon the type of component. Additional Details...

The exceptions you shared are bulletins only and always being reported as issue sending to http://node-1:8080/  node. I see all your other configuration are based off IPs.  Are all you NiFi nodes able to properly resolve "node-1" to the correct IP address?

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Back to this, again very very grateful, I made the suggested changes regarding the definition of the s2s urls, Yes, the nodes resolve well by node-1, node-2, node-3, the issue with the s2s messages is that they reach the log and fill up the disk, I have budget limitations in technology, I can not allow it to increase, then my nifi falls by disk, therefore I have maintenance shells that run every day to maintain disk usage, I checked and indeed the 2 controllers, the s2sbolletin and the s2s status, work for me, they will remain in different tables, I made the changes, I was confused with the formats regarding one with the other, thank you very much Matt

[#4]

Thanks Matt

I opened a new message because the initial context of this thread already supported me with its solution and this would be another query where I attached the properties and the flow.

[#5]

@Bern this is your new question without a accepted solution.  So a bit confused by your last response.

Matt