Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to configure Flume with Kafka channel without source ?

avatar
Contributor

It complains if the source is not specified in the flume configuration. I have another kafka producer writing to kafka topic (so I don't the source in flume configuration)  and would like just to read those messages in flume (with kafka channel and hdfs sink).

Regards

2 ACCEPTED SOLUTIONS

avatar
What version of CDH are you using? In newer versions it displays a warning, but still allows you to save the changes without a source.

-pd

View solution in original post

avatar
No, if you only have one sink, you would have one file (assuming you don't use header variable buckets). The sink will consume from all three partitions and may deliver those in one batch to one file.

-pd

View solution in original post

3 REPLIES 3

avatar
What version of CDH are you using? In newer versions it displays a warning, but still allows you to save the changes without a source.

-pd

avatar
Contributor

Hello,

 

you're right. On CDH 5.14 it's just a warning. Thanks !

 

Just one more question - if the topic in kafka channel has three partitions, is it expected that flume hdfs sink will allocate three separate files on HDFS?

 

Regards,
Nenad

avatar
No, if you only have one sink, you would have one file (assuming you don't use header variable buckets). The sink will consume from all three partitions and may deliver those in one batch to one file.

-pd