Member since
12-05-2016
1
Post
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3488 | 12-05-2016 10:51 AM |
12-05-2016
10:51 AM
1 Kudo
Hi Rafa, Sorry to hear you are having trouble with performance. I suspect you are on the right track when it comes to batch sizes, but you may need some further tuning. Could you start by posting the whole of your agent.conf (e.g. including sources and channels) as it's possible the latency is being introduced elsewhere. Also, what version of Flume/CDH are you running - the configuration of Kafka Sinks changed quite dramatically in Flume 1.7 (with the relevant Kafka bits also featuring in CDH5.8+). There's some performance tuning tips in http://blog.cloudera.com/blog/2016/08/new-in-cloudera-enterprise-5-8-flafka-improvements-for-real-time-data-ingest/ (although they are geared towards increasing throughput rather than decreasing latecy, there will be some relevant settings in there). As a bit of simple maths: if you are expecting 1-2 messages per second, with a batch size of 10, it could be waiting 5-10 seconds before a batch is received and therefore before sending on. In this instance I'd be looking to tune the batch sizes down to 1 across the board in order to ensure that messages are passed on as soon as they are received. Please give that a try, and post some more details about your config and we'll see if we can help. Tristan
... View more