Hello,   We're looking for some advice on how to identify possible improvements on our current NiFi flow that is a copy of what we had implemented in Flume, both running on AWS with CDP Private Cloud.   Our use case begins with an HTTPS listener that in its simplest form receives GET requests and processes them and writes events into Kafka.   Between those two ends, we perform some validations and transformations: - we convert the request parameters into JSON format (using 5 processors with AttributesToJSON and ReplaceText) - convert JSON format to rename properties using a JoltTransformJSON - validations using a couple of RouteOnAttribute + ReplaceText to confirm mandatory fields and transform null values - mapping of numeric values into strings using reference data (using UpdateAttribute, RouteOnAttribute and 2 LookupAttribute) - convert JSON into Avro format with ConvertRecord - returning the HTTP response + publishing each individual record to Kafka   In total, we have around 20-25 processors on the critical path and around the same to process any errors along the way and we're able to reach about 3,000 records/second using 3 16-core/16GB heap nodes (sizing recommended by Cloudera).   Before, we were reaching 9,000 records/second in Flume, having 3 identical nodes to process HTTP and validations and 3 more to receive the Avro messages and publish them to Kafka.   We've already tried switching our gp2 disks (3000 IOPS, 125MiB/s) to gp3 (3000 IOPS, 500MiB/s), putting the repositories into different volumes, but none of those translated into gains in performance.   Any ideas on how identify improvements like CPU (that only reaches 60% usage even with 128 threads), heap (currently using 16GB), disks (using what was reported above, but monitoring doesn't show them being stressed), custom processor development for our custom validations and transformations, etc?   Thank you. Best regards, Miguel ... View more