Reply
New Contributor
Posts: 5
Registered: ‎06-16-2017

Spark 2.1.0 and Kafka 0.9.0 connection problem

Hi there,

 

I just tried to setup spark 2.1.0 in cloudera cluster. I downloaded the files needed and i execute the instruction posted here https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Multiple-Spark-version-on-the-same... .

 

My application is using kafka streaming and when i try to execute with spark 2.1.0 i get the error

 

kafka.cluster.BrokerEndPoint cannot be cast to kafka.cluster.broker.

 

I searched a bit i know that is a little tricky to get the kafka versions right.

 

My sbt file:

 

"org.apache.spark" %% "spark-core" % "2.1.0",
"org.apache.spark" %% "spark-sql" % "2.1.0",
"org.apache.spark" %% "spark-streaming" % "2.1.0",
"org.apache.spark" %% "spark-streaming-kafka-0-8" % "2.1.0",
"org.apache.kafka" % "kafka-clients" % "0.8.2.1",

 

My spark-submit has --packages

org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0,org.apache.kafka:kafka-clients:0.8.2.1

 

Does someone knows what is missing? Which config is not ok?

 

Best regards.

New Contributor
Posts: 5
Registered: ‎06-16-2017

Re: Spark 2.1.0 and Kafka 0.9.0 connection problem

Just a little note, my app is working on the default setup of the cluster.

 

I manage to make it work, just remove this 2 lines from my spark2.1.0/conf/classpath.txt

 

/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/kafka-clients-0.9.0-kafka-2.0.0.jar
/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/kafka_2.10-0.9.0-kafka-2.0.0.jar

 

 

Best regards.

Announcements