04-17-2017 05:45 AM
I need to consume data from a quite old version of Kafka (0.7) using Flume 5.10.1. From what I read, the newer versions of CDH 5 cannot consume from a version of Kafka which is that old.
Part of our upgrade path also involves running the newest version of Kafka, so what I've been trying to research is the best way to get the data from K 0.7 to HDFS via Flume. It needs to be Flume because I'm using Morphlines for Interceptor processing, SOLR indexing and finally HDFS storage.
One thing I have been wondering is if there's a way to subscribe Kafka 2 to a Kafka 0.7 topic? I've wondered if there's a way to forward messages from an older version of Kafka to newer. Maybe there's a different solution which doesn't include any of these things such as directly consuming from Kafka 0.7 and dumping into a spooldir (less ideal).
I really haven't found anything conclusive after searching around the web and forums so any guidance would be appreciated.