Support Questions

Find answers, ask questions, and share your expertise

Spark 2.1.0 and Kafka 0.9.0 connection problem

New Contributor

Hi there,


I just tried to setup spark 2.1.0 in cloudera cluster. I downloaded the files needed and i execute the instruction posted here .


My application is using kafka streaming and when i try to execute with spark 2.1.0 i get the error


kafka.cluster.BrokerEndPoint cannot be cast to


I searched a bit i know that is a little tricky to get the kafka versions right.


My sbt file:


"org.apache.spark" %% "spark-core" % "2.1.0",
"org.apache.spark" %% "spark-sql" % "2.1.0",
"org.apache.spark" %% "spark-streaming" % "2.1.0",
"org.apache.spark" %% "spark-streaming-kafka-0-8" % "2.1.0",
"org.apache.kafka" % "kafka-clients" % "",


My spark-submit has --packages



Does someone knows what is missing? Which config is not ok?


Best regards.


New Contributor

Just a little note, my app is working on the default setup of the cluster.


I manage to make it work, just remove this 2 lines from my spark2.1.0/conf/classpath.txt





Best regards.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.