Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to set kafka client version to 0.11 when spark submit to yarn cluster?

Highlighted

how to set kafka client version to 0.11 when spark submit to yarn cluster?

New Contributor
I have executed spark job which produces messages to kafka cluster(version 0.11) on cdh 5.14 yarn.

but that job tried to connect by kafka client 0.9.

is there way that spark job connect to kafka by kafka client 0.11 in yarn cluster mode?
3 REPLIES 3
Highlighted

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

Expert Contributor

There are a couple ways you can set the spark kafka version.  Please find instructions here: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_kafka.html#running_jobs

Highlighted

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

New Contributor
Now, I can only choose up to 0.10 by SPARK_KAFKA_VERSION configuration. Can I choose up to kafka 0.11 if I upgrade CDK Kafka version?

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

Expert Contributor

You can use the .10 spark consumer version with kafka .10 and newer.

Don't have an account?
Coming from Hortonworks? Activate your account here