Reply
Highlighted
Explorer
Posts: 16
Registered: ‎09-25-2017
Accepted Solution

How to configure Flume with Kafka channel without source ?

It complains if the source is not specified in the flume configuration. I have another kafka producer writing to kafka topic (so I don't the source in flume configuration)  and would like just to read those messages in flume (with kafka channel and hdfs sink).

Regards

Cloudera Employee
Posts: 277
Registered: ‎01-09-2014

Re: How to configure Flume with Kafka channel without source ?

What version of CDH are you using? In newer versions it displays a warning, but still allows you to save the changes without a source.

-pd
Explorer
Posts: 16
Registered: ‎09-25-2017

Re: How to configure Flume with Kafka channel without source ?

Hello,

 

you're right. On CDH 5.14 it's just a warning. Thanks !

 

Just one more question - if the topic in kafka channel has three partitions, is it expected that flume hdfs sink will allocate three separate files on HDFS?

 

Regards,
Nenad

Cloudera Employee
Posts: 277
Registered: ‎01-09-2014

Re: How to configure Flume with Kafka channel without source ?

No, if you only have one sink, you would have one file (assuming you don't use header variable buckets). The sink will consume from all three partitions and may deliver those in one batch to one file.

-pd
Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.