I am a new user of nifi trying to integrate it for one of our use case. I have set up 3 node nifi cluster (26 cores, 128GB ram). I am stress testing the flow I have and not able to achieve the throughput I require. I am following the best practices articles. But with the simple flow I have I am not able to scale up more than 7Mbps transfer. Can anyone suggest me how to increase the throughput ? Flow: [GenerateFlowfile (Primary node only) -> updateAttributes -> processGroupPort] -> OuputPutPort
(1kb messagesin chunks of 10)
[RPG] -> [inputPortProcessGroup -> UpdateAttributes -> drop] Flowfiles are stuck transferring between processGroupPort -> OutputPort.
I have tried with various combination of backpressure, number of threads and batch size in RPG. The maximum I could achive was 8Mbps. I have seen various usecases where users have achived throughput much more than this. Few settings I have changed in nifi.conf nifi.queue.swap.threshold=120000 backpressure threshold = 10000 java heap = 20GB
Maximum Timer Driven Thread Count in controller settings : 500 Can you guys please help me configure the flow for optimum performance ? Thanks
... View more