Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to configure Flume with Kafka channel without source ?

SOLVED Go to solution

How to configure Flume with Kafka channel without source ?

Explorer

It complains if the source is not specified in the flume configuration. I have another kafka producer writing to kafka topic (so I don't the source in flume configuration)  and would like just to read those messages in flume (with kafka channel and hdfs sink).

Regards

2 ACCEPTED SOLUTIONS

Accepted Solutions

Re: How to configure Flume with Kafka channel without source ?

Super Collaborator
What version of CDH are you using? In newer versions it displays a warning, but still allows you to save the changes without a source.

-pd
Highlighted

Re: How to configure Flume with Kafka channel without source ?

Super Collaborator
No, if you only have one sink, you would have one file (assuming you don't use header variable buckets). The sink will consume from all three partitions and may deliver those in one batch to one file.

-pd
3 REPLIES 3

Re: How to configure Flume with Kafka channel without source ?

Super Collaborator
What version of CDH are you using? In newer versions it displays a warning, but still allows you to save the changes without a source.

-pd

Re: How to configure Flume with Kafka channel without source ?

Explorer

Hello,

 

you're right. On CDH 5.14 it's just a warning. Thanks !

 

Just one more question - if the topic in kafka channel has three partitions, is it expected that flume hdfs sink will allocate three separate files on HDFS?

 

Regards,
Nenad

Highlighted

Re: How to configure Flume with Kafka channel without source ?

Super Collaborator
No, if you only have one sink, you would have one file (assuming you don't use header variable buckets). The sink will consume from all three partitions and may deliver those in one batch to one file.

-pd