Reply
Expert Contributor
Posts: 119
Registered: ‎07-01-2015

KafkaConsumer.subscribe 0.9 vs 0.10 in Structured streaming

Hi I am trying to run a structured streaming on CDH 5.11.1 with Kafka 2.2.0-1.2.2.0.p0.68.

The streaming fails that there is no subscribe method with this definition:

 

org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)

 

Found out that this is due the change between 0.9 and 0.10 version. 

My project contains a sbt pointing to a 0.10 artifacts,

libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
libraryDependencies += "org.apache.spark" % "spark-sql-kafka-0-10_2.11" % "2.2.0"
libraryDependencies += "org.apache.kafka" % "kafka-clients" % "0.10.2.1"

 

the CDH should use also Kafka 0.10. The spark-submit job has an explicit parameter:

 

spark-submit ... --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0,org.apache.kafka:kafka-clients:0.10.2.1

 

so I dont understand where the 0.9 comes from, and why it expects the older API.

 

 

17/09/20 14:48:07 ERROR streaming.StreamExecution: Query [id = 60109f59-c6fc-4956-b559-16b41570e45a, runId = 8c86c4a5-f510-4269-a33e-fb19390cb054] terminated with error
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V
at org.apache.spark.sql.kafka010.SubscribeStrategy.createConsumer(ConsumerStrategy.scala:63)

 

Announcements