question Re: How to configure Flume with Kafka channel without source ? in Support Questions

How to configure Flume with Kafka channel without source ?

NVidak — Fri, 16 Sep 2022 14:18:06 GMT

It complains if the source is not specified in the flume configuration. I have another kafka producer writing to kafka topic (so I don't the source in flume configuration) and would like just to read those messages in flume (with kafka channel and hdfs sink).

Regards

Re: How to configure Flume with Kafka channel without source ?

pdvorak — Wed, 10 Apr 2019 15:49:01 GMT

What version of CDH are you using? In newer versions it displays a warning, but still allows you to save the changes without a source.

-pd

Re: How to configure Flume with Kafka channel without source ?

NVidak — Thu, 11 Apr 2019 06:18:44 GMT

Hello,

you're right. On CDH 5.14 it's just a warning. Thanks !

Just one more question - if the topic in kafka channel has three partitions, is it expected that flume hdfs sink will allocate three separate files on HDFS?

Regards,
Nenad

Re: How to configure Flume with Kafka channel without source ?

pdvorak — Wed, 01 May 2019 15:07:59 GMT

No, if you only have one sink, you would have one file (assuming you don't use header variable buckets). The sink will consume from all three partitions and may deliver those in one batch to one file.

-pd