05-20-2016 05:03 AM
Ah sorry we are back to run-example. It should work. I just tried it on my CDH 5.7 cluster with Kafka 0.9 and it works as expected. Are you sure you don't have old libs, parcels, config left around?
05-27-2016 11:59 AM
Does this imply we have to upgrade Kafka broker version to 0.9.0.0 as well as Kafka dependency on the Spark side to use 0.9.0.0 ?
We are directly using kafka through Spark streaming and running into following exception on switching to CDH 5.7.0. I updated the pom of my spark app to use 0.9.0.0 but that did not help.
We were running Spark 1.6.0 and Kafka 0.8.2.1 and are just moving to CDH 5.7.0. Upgrading Kafka to 0.9.0.0 would be the last thing we want to do since it requires moving all the components in our system to use that version. So in other words, is it true in order to use CDH 5.7.0, Kafka brokers have to be at 0.9.0.0 ?
05-27-2016 12:59 PM
Yes it's my understanding that to use Spark Streaming with Kafka in CDH 5.7, you will need to run Kafka 0.9 (aka the 2.x parcel from Cloudera): http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_kafka_ki.html
I think the motivation was that this is required to have any hope of enabling security for Kafka, and because of 0.8/0.9 incompatibilities. 0.8 is a bit old at this stage too.