question Re: Is Flume to Kafka Sink supported in HDP 2.3.2? in Archives of Support Questions (Read Only)

Is Flume to Kafka Sink supported in HDP 2.3.2?

raghavendran_c — Tue, 12 Jan 2016 02:56:20 GMT

Hi,

the following link says Kafka Source & Kafka Sink are supported in the Flume that comes with HDP2.3.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_HDP_RelNotes/content/new-features-230.html

But the JIRA (https://issues.apache.org/jira/browse/FLUME-2242) mentioned in the hortonworks docs says that it is available only from Flume 1.6 (while HDP 2.3.2 has only Flume 1.5.2).

Can someone confirm that Kafka Source & kafka Sink is indeep available in the Flume that comes with HDP 2.3.2 ?

thanks,

Raga

Re: Is Flume to Kafka Sink supported in HDP 2.3.2?

cnormile — Tue, 12 Jan 2016 02:58:15 GMT

Kafka Source and Kafka Sink are supported in the Flume 1.5.2 Build shipped with HDP 2.3+

Re: Is Flume to Kafka Sink supported in HDP 2.3.2?

vance_wei — Tue, 19 Jan 2016 01:11:59 GMT

I have scanned all the java classes from the flume 1.5.2 lib, and I have two jars relating to kafka, the kafka-client-0.8.2-beta.jar and kafka_2.10-0.8.2-beta.jar. However, they don't seem to be for source and sink. The flume 1.6 contains classes for kafka source and sink explicitly.

Therefore, I doubt the source and sink works with Flume 1.5.2 in HDP 2.3+. I am also looking into ways to get Flume to work with Kafka in HDP 2.3. If any of you have working examples, and can share your conf file, I would highly appreciate it.

Re: Is Flume to Kafka Sink supported in HDP 2.3.2?

aervits — Tue, 19 Jan 2016 01:26:18 GMT

@Raghavendran Chellappa the release notes for HDP are the source of truth, Jira is external to Hortonworks. If Release notes say we support it, then that's the way to go. What is it that you find not working? We have Kafka 0.8.2.0 stable not beta. Kafka 0.9 also works. We usually backport critical features. Also, have you tried Nifi, it has all the latest Kafka support including Kerberos.