Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to set kafka client version to 0.11 when spark submit to yarn cluster?

how to set kafka client version to 0.11 when spark submit to yarn cluster?

New Contributor
I have executed spark job which produces messages to kafka cluster(version 0.11) on cdh 5.14 yarn.

but that job tried to connect by kafka client 0.9.

is there way that spark job connect to kafka by kafka client 0.11 in yarn cluster mode?
3 REPLIES 3

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

Expert Contributor

There are a couple ways you can set the spark kafka version.  Please find instructions here: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_kafka.html#running_jobs

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

New Contributor
Now, I can only choose up to 0.10 by SPARK_KAFKA_VERSION configuration. Can I choose up to kafka 0.11 if I upgrade CDK Kafka version?
Highlighted

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

Expert Contributor

You can use the .10 spark consumer version with kafka .10 and newer.