Support Questions
Find answers, ask questions, and share your expertise

Flume memory transaction error and performance


We got the below error :

018-09-17 12:09:32,303 ERROR org.apache.flume.source.kafka.KafkaSource: KafkaSource EXCEPTION, {} org.apache.flume.ChannelException: Put queue for MemoryTransaction of capacity 100 full, consider committing more frequently, increasing capacity or increasing thread count

We updated the below parameters :

flume1.sources.kafka1.batchSize = 500

flume1.channels.channel1.capacity = 10000

flume1.channels.channel1.transactionCapacity = 1000

The error seems to have gone away. So is this good or anything else needs to be done?

Also what other parameters are relevant for performance and capacity.

Appreciate the feedback.