Support Questions
Find answers, ask questions, and share your expertise

Spark streaming batch size spikes suddenly. I enabled the backpressure too. Spark version : 1.6.3

Spark streaming batch size spikes suddenly. I enabled the backpressure too. Spark version : 1.6.3

New Contributor

I am streaming data from kafka and trying to limit the number of events per batch to 10 events. After processing for 10-15 batches, there is a sudden spike in the batch size. Attached are my settings:

spark.streaming.kafka.maxRatePerPartition=1

spark.streaming.backpressure.enabled=true

spark.streaming.backpressure.pid.minRate=1

spark.streaming.receiver.maxRate=2

43620-events.png