@Threepwood ,

What you're proposing here is something called Stateless NiFi execution.

In a normal NiFi flow data is always shared in the form of flowfiles and there's no way to work around that. I/O is a tax that NiFi pays to make it more flexible. It's know to be I/O heavy but it still performs very well and can handle huge volumes of data if you follow the best practices when building your flows.

Stateless NiFi execution can be used for a subset of NiFi flows that don't need to store state. In a NiFi Stateless execution, nothing is written to disk. Data is sent from one processor to another through function calls. It can achieve much faster performance this way, but it's limited to stateless flows.

Mark Payne has another video where he talks about this and how to use it to achieve Exactly-Once delivery from Kafka to Kafka: https://www.youtube.com/watch?v=VyzoD8eh-t0

Cheers,

André