How to configure Flume with Kafka channel without source ?

[#1]

It complains if the source is not specified in the flume configuration. I have another kafka producer writing to kafka topic (so I don't the source in flume configuration)  and would like just to read those messages in flume (with kafka channel and hdfs sink).

Regards

[#2 · Accepted solution]

What version of CDH are you using? In newer versions it displays a warning, but still allows you to save the changes without a source.

-pd

[#3 · Accepted solution]

What version of CDH are you using? In newer versions it displays a warning, but still allows you to save the changes without a source.

-pd

[#4]

Hello,

you're right. On CDH 5.14 it's just a warning. Thanks !

Just one more question - if the topic in kafka channel has three partitions, is it expected that flume hdfs sink will allocate three separate files on HDFS?

Regards,
Nenad

[#5 · Accepted solution]

No, if you only have one sink, you would have one file (assuming you don't use header variable buckets). The sink will consume from all three partitions and may deliver those in one batch to one file.

-pd