Reply
New Contributor
Posts: 2
Registered: ‎08-01-2018

how to set kafka client version to 0.11 when spark submit to yarn cluster?

I have executed spark job which produces messages to kafka cluster(version 0.11) on cdh 5.14 yarn.

but that job tried to connect by kafka client 0.9.

is there way that spark job connect to kafka by kafka client 0.11 in yarn cluster mode?
Highlighted
Cloudera Employee
Posts: 97
Registered: ‎05-10-2016

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

There are a couple ways you can set the spark kafka version.  Please find instructions here: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_kafka.html#running_jobs

New Contributor
Posts: 2
Registered: ‎08-01-2018

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

Now, I can only choose up to 0.10 by SPARK_KAFKA_VERSION configuration. Can I choose up to kafka 0.11 if I upgrade CDK Kafka version?
Cloudera Employee
Posts: 97
Registered: ‎05-10-2016

Re: how to set kafka client version to 0.11 when spark submit to yarn cluster?

You can use the .10 spark consumer version with kafka .10 and newer.

Announcements