Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

sbt compilation error

avatar
Super Collaborator

utilitiesscala.txtprinttweetsscala.txtmy build file is ,I am attaching the source code .

[root@hadoop1 SparkStreaming]# more build.sbt
name := "SparkStreaming"
version := "1.0"
scalaVersion := "2.11.7"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % "1.2.1","org.apache.spark" % "spark-mllib_2.10" % "1.2.1")
scalacOptions += "-deprecation"

compilation error :

[root@hadoop1 SparkStreaming]# sbt compile
[info] Set current project to SparkStreaming (in build file:/root/SparkStreaming/)
[info] Compiling 2 Scala sources to /root/SparkStreaming/target/scala-2.11/classes...
[error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:7: object twitter is not a member of package org.apache.spark.streaming
[error] import org.apache.spark.streaming.twitter._
[error]                                   ^
[error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:29: not found: value TwitterUtils
[error]     val tweets = TwitterUtils.createStream(ssc, None)
[error]                  ^
[error] two errors found
[error] (compile:compile) Compilation failed
[error] Total time: 3 s, completed Sep 15, 2016 12:57:15 PM

thanks

1 ACCEPTED SOLUTION

avatar
Super Collaborator

this build.sbt fix the issue and now it compiles the package fine

[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

[root@hadoop1 TwitterPopularTags]#

View solution in original post

2 REPLIES 2

avatar
Super Collaborator

the twitter related jars I have placed in the 'resource' folder as shown below :

[root@hadoop1 resources]# pwd /root/SparkStreaming/src/main/resources [root@hadoop1 resources]# ls dstream-twitter_2.11-0.1.0-SNAPSHOT.jar target twitter4j-core-4.0.4.ziptwitter4j-core-4.0.4.jar twitter4j-stream-4.0.4.jar [root@hadoop1 resources]# jar tvf dstream-twitter_2.11-0.1.0-SNAPSHOT.jar | grep TwitterUtils 5862 Wed Aug 10 11:11:36 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils$.class 4749 Wed Aug 10 11:11:34 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils.class [root@hadoop1 resources]#

avatar
Super Collaborator

this build.sbt fix the issue and now it compiles the package fine

[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

[root@hadoop1 TwitterPopularTags]#