Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark 2.1.0 and Kafka 0.9.0 connection problem

Spark 2.1.0 and Kafka 0.9.0 connection problem

New Contributor

Hi there,

 

I just tried to setup spark 2.1.0 in cloudera cluster. I downloaded the files needed and i execute the instruction posted here https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Multiple-Spark-version-on-the-same... .

 

My application is using kafka streaming and when i try to execute with spark 2.1.0 i get the error

 

kafka.cluster.BrokerEndPoint cannot be cast to kafka.cluster.broker.

 

I searched a bit i know that is a little tricky to get the kafka versions right.

 

My sbt file:

 

"org.apache.spark" %% "spark-core" % "2.1.0",
"org.apache.spark" %% "spark-sql" % "2.1.0",
"org.apache.spark" %% "spark-streaming" % "2.1.0",
"org.apache.spark" %% "spark-streaming-kafka-0-8" % "2.1.0",
"org.apache.kafka" % "kafka-clients" % "0.8.2.1",

 

My spark-submit has --packages

org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0,org.apache.kafka:kafka-clients:0.8.2.1

 

Does someone knows what is missing? Which config is not ok?

 

Best regards.

1 REPLY 1

Re: Spark 2.1.0 and Kafka 0.9.0 connection problem

New Contributor

Just a little note, my app is working on the default setup of the cluster.

 

I manage to make it work, just remove this 2 lines from my spark2.1.0/conf/classpath.txt

 

/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/kafka-clients-0.9.0-kafka-2.0.0.jar
/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/kafka_2.10-0.9.0-kafka-2.0.0.jar

 

 

Best regards.