I am streaming data from kafka and trying to limit the number of events per batch to 10 events. After processing for 10-15 batches, there is a sudden spike in the batch size. Attached are my settings: spark.streaming.kafka.maxRatePerPartition=1 spark.streaming.backpressure.enabled=true spark.streaming.backpressure.pid.minRate=1 spark.streaming.receiver.maxRate=2 ... View more