So GET requests work fine for you but POSTs fail? Perhaps your user does not have write permissions? I recommend opening devtools and looking at the Network tab to see what requests get sent when you do certain operations (starting/stopping a process group) in the UI. You should be able to replicate them with the REST API if you can do them manually in the UI. ... View more

Hi @MuruganFinastra , Are you sure you are attaching your token correctly to your requests? You should be using an Authorization header with a value of "Bearer <token>" for all requests once you fetch the token. ... View more

Hey @eylon , did you ever find a solution to this problem? We are facing the exact same issue right now where we'd like to write to elastic without records due to a changing schema, yet our files contain batches of jsons and so we can't use PutElasticsearchJson as we'd like. ... View more

I am also experiencing this issue when attempting to write data to a redis in cluser mode. Did you find a solution or workaround @sofronic ? ... View more

Hey Samuel,   Perhaps the source of my own confusion is that I do not know how Apache Benchmark works.   Considering only a few hundred requests get dropped (out of hundreds of thousands), this would mean nifi is indeed able to ingest (almost) all the data you send it. As such, I think we can rule out the load balancer and ListenHttpRequest processor.   You expressed that you don't believe any Back Pressure is applied in the flow because the queues between processors are configured to contain 20,000 flowfiles at once whilst you only send about 5000 concurrent requests (this also confuses me - 5000 concurrent requests, but at what rate? every second?). The way you described this sounds odd to me - whether a flow has any backpressure is not a function of its configuration and the data ingestion rate. Rather, I expected you would visually examine the flow as you run your benchmark test and check whether at any point in the flow the queues fill up and turn red. Could you verify that this is how you checked for backpressure?   If you claim that no queues get filled up in nifi and no requests get dropped, I can't think of an explanation for how the req/s rate changes. No Back Pressure implies the flow is never under duress, and no requests dropped means that all requests that get sent are processed. Again, perhaps my lack of understanding is due to not knowing how Apache Benchmark works.   I believe if you'll clarify these points for me we could get much closer to understanding what the source of the issue is 🙂   Regards, Eyal ... View more

Hey @F_Amini,   Has your issue been resolved? I'd be glad to help if needed.  ... View more

Hello Samuel,   I would like to verify- As you test your flow, do the queues fill up and cause any backpressure?   What is unclear to me is whether the bad performance originates from: the flow not running well (queues begin to fill up which then causes up-stream processors to stop getting scheduled, etc.) the initial HandleHttpRequest processor not being capable of handling ingestion (and so the flow looks healthy but requests get dropped before being turned into flowfiles).   If the first case is true, I'd like to know where the backpressure in the flow originates from.   If the second case is true, I've found this post where a couple of nifi experts advise how to increase http ingestion.   As a side thought - is it possible your load balancer can't keep up with the throughput?   My regards, Eyal   ... View more

So long as backpressure is building up behind a processor, the processor that is previous in line will not be scheduled. This will fall back all the way throughout your flow until it reaches the processor that fetches the data and cause it to stop. As such, all it takes is one weak-link processor to limit your entire flow's throughput. I see the relationship coming out of your HashContent is also red, which implies that there is a different processor further down-flow that is causing backpressure. Can you describe which processor causes this? perhaps fixing its slow performance will improve your entire flow. ... View more

That's correct. I believe my earlier tests were failing because I used the "inherit record schema" setting, which obviously wouldn't work on the writer if the schema changes after forking. Here is an avro schema I generated that should describe the output exactly as @Fredi described (generated using this website), however even when using this schema in my record writer, the output still comes out empty. { "name": "MyClass", "type": "record", "namespace": "myNamespace", "fields": [ { "name": "Key1", "type": { "type": "array", "items": { "name": "Key1_record", "type": "record", "fields": [ { "name": "NestedKey1", "type": "string" }, { "name": "NestedKey2", "type": "string" }, { "name": "NestedKey3", "type": "string" } ] } } } ] }   I believe at this point the challenge is simply writing an accurate avro schema for the output data. ... View more