Support Questions
Find answers, ask questions, and share your expertise

how to set kafka client version to 0.11 when spark submit to yarn cluster?

New Contributor
I have executed spark job which produces messages to kafka cluster(version 0.11) on cdh 5.14 yarn.

but that job tried to connect by kafka client 0.9.

is there way that spark job connect to kafka by kafka client 0.11 in yarn cluster mode?
3 REPLIES 3

Expert Contributor

There are a couple ways you can set the spark kafka version.  Please find instructions here: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_kafka.html#running_jobs

New Contributor
Now, I can only choose up to 0.10 by SPARK_KAFKA_VERSION configuration. Can I choose up to kafka 0.11 if I upgrade CDK Kafka version?

Expert Contributor

You can use the .10 spark consumer version with kafka .10 and newer.

; ;