Is Flume to Kafka Sink supported in HDP 2.3.2?

[#1]

Hi,

the following link says Kafka Source & Kafka Sink are supported in the Flume that comes with HDP2.3.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_HDP_RelNotes/content/new-features-230.htm...

But the JIRA (https://issues.apache.org/jira/browse/FLUME-2242) mentioned in the hortonworks docs says that it is available only from Flume 1.6 (while HDP 2.3.2 has only Flume 1.5.2).

Can someone confirm that Kafka Source & kafka Sink is indeep available in the Flume that comes with HDP 2.3.2 ?

thanks,

Raga

[#2 · Accepted solution]

Kafka Source and Kafka Sink are supported in the Flume 1.5.2 Build shipped with HDP 2.3+

[#3]

I have scanned all the java classes from the flume 1.5.2 lib, and I have two jars relating to kafka, the kafka-client-0.8.2-beta.jar and kafka_2.10-0.8.2-beta.jar. However, they don't seem to be for source and sink. The flume 1.6 contains classes for kafka source and sink explicitly.

Therefore, I doubt the source and sink works with Flume 1.5.2 in HDP 2.3+. I am also looking into ways to get Flume to work with Kafka in HDP 2.3. If any of you have working examples, and can share your conf file, I would highly appreciate it.

[#4]

@Raghavendran Chellappa the release notes for HDP are the source of truth, Jira is external to Hortonworks. If Release notes say we support it, then that's the way to go. What is it that you find not working? We have Kafka 0.8.2.0 stable not beta. Kafka 0.9 also works. We usually backport critical features. Also, have you tried Nifi, it has all the latest Kafka support including Kerberos.