NiFi — ye olde "destination does not accept flowfiles . . . processor configured to delivery flowfiles"

[#1]

We have a number of PostHTTP processors sending files to ListenHTTP processors that chug along day after day happy as a clam. Then all of a sudden, there will be a spate of these errors: "Cannot send . . . to . . . because the destination does not accept FlowFiles and this processor is configured to delivery FlowFiles; routing to failure."

For one of the PostHTTP processors, I created additional routes on success/failure so that I could look at and compare the files. These files are JSON, and after downloading the files to both success and failure, both are correctly formed JSON. We have a retry-loop flow here, too, that on failure retries the post five times. So these additional routes also allowed me to see in provenance one of the files that failed with the above error eventually went through.

I'm looking for any insight on why this might be happening.

About = niagarafiles 11.31.0 based on Apache NiFi 1.26.0a

[#2]

@noncitizen 

Can you share your PostHTTP and ListenHTTP processor configurations and scheduling?
What is the volume of FlowFiles queued to the PostHTTP?
How many PostHTTP processors sending to same ListenHTTP?
Does ListenHTTP outbound connection queue fill resulting in backpressure being applied on the ListenHTTP?

Since this is a sporadic issue, trying to get a better idea of setup and conditions at time of issue.

Thanks,

Matt

[#3]

This is on an isolated system, so screenshots are, well, not impossible, but very difficult.

PostHTTP (if not listed, it has "No value set"):

• URL = our url/contentListener
• Max Batch Size = 100 MB
• SSL Context Service = StandardRestrictedSSLContextService
• Send a FlowFile = true
• Compression Level = 0
• Connection Timeout = 360 sec
• Data Timeout = 360 sec
• User Agent = Apache-HttpClient/4.5.5 (Java/1.8.0_262)
• Content-Type = ${mime.type}
• Disable Connection Reuse = true

ListenHTTP:

• Base Path = contentListener
• Listening Port = 9443
• SSL Context Service = StandardRestrictedSSLContextService
• HTTP Protocols = h2 http/1.1
• Client Authentication = WANT
• Authorized Subject DN Pattern = .*
• Authorized Issuer DN Pattern = .*
• Max Unconfirmed Flowfile Time = 300 secs
• HTTP Headers to receive as Attributes (Regex) = .*
• Return Code = 200
• Multipart Request Max Size = 15GB
• Multipart Read Buffer Size = 1MB
• Maximum Threadpool Size = 400
• Request Time Out = 30 secs

This particular ListenHTTP receives from both PostHTTP processors in the same instance, in other instances we own, and from many external sources. Yes, it gets hammered all day long.

Under normal circumstances, there is no back pressure on the ListenHTTP processor. It only occurs if there is some bottleneck downstream because of some condition not handled gracefully that might cause queues to back up, but that has occurred very rarely.

Thanks for your input!