Support Questions

Find answers, ask questions, and share your expertise

sbt compilation error

avatar
Super Collaborator

utilitiesscala.txtprinttweetsscala.txtmy build file is ,I am attaching the source code .

[root@hadoop1 SparkStreaming]# more build.sbt
name := "SparkStreaming"
version := "1.0"
scalaVersion := "2.11.7"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % "1.2.1","org.apache.spark" % "spark-mllib_2.10" % "1.2.1")
scalacOptions += "-deprecation"

compilation error :

[root@hadoop1 SparkStreaming]# sbt compile
[info] Set current project to SparkStreaming (in build file:/root/SparkStreaming/)
[info] Compiling 2 Scala sources to /root/SparkStreaming/target/scala-2.11/classes...
[error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:7: object twitter is not a member of package org.apache.spark.streaming
[error] import org.apache.spark.streaming.twitter._
[error]                                   ^
[error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:29: not found: value TwitterUtils
[error]     val tweets = TwitterUtils.createStream(ssc, None)
[error]                  ^
[error] two errors found
[error] (compile:compile) Compilation failed
[error] Total time: 3 s, completed Sep 15, 2016 12:57:15 PM

thanks

1 ACCEPTED SOLUTION

avatar
Super Collaborator

this build.sbt fix the issue and now it compiles the package fine

[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

[root@hadoop1 TwitterPopularTags]#

View solution in original post

2 REPLIES 2

avatar
Super Collaborator

the twitter related jars I have placed in the 'resource' folder as shown below :

[root@hadoop1 resources]# pwd /root/SparkStreaming/src/main/resources [root@hadoop1 resources]# ls dstream-twitter_2.11-0.1.0-SNAPSHOT.jar target twitter4j-core-4.0.4.ziptwitter4j-core-4.0.4.jar twitter4j-stream-4.0.4.jar [root@hadoop1 resources]# jar tvf dstream-twitter_2.11-0.1.0-SNAPSHOT.jar | grep TwitterUtils 5862 Wed Aug 10 11:11:36 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils$.class 4749 Wed Aug 10 11:11:34 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils.class [root@hadoop1 resources]#

avatar
Super Collaborator

this build.sbt fix the issue and now it compiles the package fine

[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

[root@hadoop1 TwitterPopularTags]#