Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka/Spark Streaming integration code - compilation using sbt failing

Highlighted

Kafka/Spark Streaming integration code - compilation using sbt failing

Expert Contributor

I'm trying to compile kafka & Spark Streaming integration code i.e. reading from Kafka using Spark Streaming, and the sbt build is failing with error

- [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found

Scala version -> 2.10.7

Spark Version -> 2.1.0

Kafka version -> 0.9

sbt version -> 0.13

Contents of sbt files is as shown below ->

1) vi spark_kafka_code/project/plugins.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

2) vi spark_kafka_code/sparkkafka.sbt

import AssemblyKeys._ assemblySettings

name := "SparkKafka Project"

version := "1.0" scalaVersion := "2.11.7"

val sparkVers = "2.1.0"

// Base Spark-provided dependencies

libraryDependencies ++= Seq(

"org.apache.spark" %% "spark-core" % sparkVers % "provided",

"org.apache.spark" %% "spark-streaming" % sparkVers % "provided",

"org.apache.spark" %% "spark-streaming-kafka" % sparkVers)

mergeStrategy in assembly := {

case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard

case m if m.toLowerCase.startsWith("META-INF") => MergeStrategy.discard

case "reference.conf" => MergeStrategy.concat

case m if m.endsWith("UnusedStubClass.class") => MergeStrategy.discard

case _ => MergeStrategy.first

}

i launch sbt, and then try to create an eclipse project, complete error is as shown below - Any ideas on what the issue is ?

------------------------------------------------------------------------------------------ >

sbt [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins [info] Loading project definition from /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/project [info] Set current project to SparkKafka Project (in build file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/) > eclipse [info] About to create Eclipse project files for your project(s). [info] Updating {file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/}spark_kafka_code... [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... [warn] module not found: org.apache.spark#spark-streaming-kafka_2.11;2.1.0 [warn] ==== local: tried [warn] /Users/karanalang/.ivy2/local/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml [warn] ==== activator-launcher-local: tried [warn] /Users/karanalang/.activator/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml [warn] ==== activator-local: tried [warn] /Users/karanalang/Documents/Technology/SCALA/activator-dist-1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml [warn] ==== public: tried [warn] https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kaf... [warn] ==== typesafe-releases: tried [warn] http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-s... [warn] ==== typesafe-ivy-releasez: tried [warn] http://repo.typesafe.com/typesafe/ivy-releases/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivy... [info] Resolving jline#jline;2.12.1 ... [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note: Unresolved dependencies path: [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 (/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/sparkkafka.sbt#L12-16) [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 [trace] Stack trace suppressed: run last *:update for the full output. [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found [info] Updating {file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/}spark_kafka_code... [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... [warn] module not found: org.apache.spark#spark-streaming-kafka_2.11;2.1.0 [warn] ==== local: tried [warn] /Users/karanalang/.ivy2/local/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml [warn] ==== activator-launcher-local: tried [warn] /Users/karanalang/.activator/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml [warn] ==== activator-local: tried [warn] /Users/karanalang/Documents/Technology/SCALA/activator-dist-1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml [warn] ==== public: tried [warn] https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kaf... [warn] ==== typesafe-releases: tried [warn] http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-s... [warn] ==== typesafe-ivy-releasez: tried [warn] http://repo.typesafe.com/typesafe/ivy-releases/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivy... [info] Resolving jline#jline;2.12.1 ... [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note: Unresolved dependencies path: [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 (/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/sparkkafka.sbt#L12-16) [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 [trace] Stack trace suppressed: run last *:update for the full output. [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found [error] Could not create Eclipse project files: [error] Error evaluating task 'scalacOptions': error [error] Error evaluating task 'externalDependencyClasspath': error >

2 REPLIES 2

Re: Kafka/Spark Streaming integration code - compilation using sbt failing

Expert Contributor

@Bernhard Walter, @mqureshi @Kuldeep Kulkarni - looping you in, any ideas on this ?

Re: Kafka/Spark Streaming integration code - compilation using sbt failing

Expert Contributor

@Bernhard Walter, @Kuldeep Kulkarni, @mqureshi . here is the update on this..

i was able to get this working. here is what i added in build.sbt file ----------------------------------------------------------------------------------------------

scalaVersion := "2.11.7"

val sparkVers = "2.1.0"

// Base Spark-provided dependencies

libraryDependencies ++= Seq(

"org.apache.spark" %% "spark-core" % sparkVers % "provided",

"org.apache.spark" %% "spark-streaming" % sparkVers % "provided",

"org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % sparkVers)

------------------------------------------------------------------------------------------------

However, i'm running into addition issue when compiling the file using sbt .. it gives errors as shown below :

Pls note - I've added the jars in eclipse, and it works .. however when i use sbt to compile, it is failing.

What needs to be done ? I've also tried added the jars in CLASSPATH, but still get the same error.

-------------------------------------------------------------------------------------------------

[info] Compiling 1 Scala source to /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/target/scala-2.11/classes...

[error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:6: object SQLContext is not a member of package org.apache.spark.sql

[error] import org.apache.spark.sql.SQLContext

[error] ^

[error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:10: object SparkSession is not a member of package org.apache.spark.sql

[error] import org.apache.spark.sql.SparkSession

[error] ^

[error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:11: object types is not a member of package org.apache.spark.sql

[error] import org.apache.spark.sql.types.StructField

[error] ^

[error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:12: object types is not a member of package org.apache.spark.sql

[error] import org.apache.spark.sql.types.StringType

[error] ^

[error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:13: object types is not a member of package org.apache.spark.sql

[error] import org.apache.spark.sql.types.StructType

[error] ^

[error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:14: object Row is not a member of package org.apache.spark.sql

[error] import org.apache.spark.sql.Row

[error] ^

Don't have an account?
Coming from Hortonworks? Activate your account here