12-20-2017 06:10 PM
I am learning kafka - spark streaming and trying to build a small unreal setup as follows:
i have observed many batches submitted in kafka consumer log while writing the data to kafka topic. My question is: will it submit a batch for each record? i havd 70000 records in the file. will it submit 70000 batches to read data from kafka topic? is there a limitation on # of batches to be submitted? can i batch 10000 recs into one batch & write it to kafka topic?