question Re: sbt compilation error in Archives of Support Questions (Read Only)

sbt compilation error

aliyesami — Fri, 16 Sep 2016 00:05:06 GMT

utilitiesscala.txt printtweetsscala.txtmy build file is ,I am attaching the source code .

[root@hadoop1 SparkStreaming]# more build.sbt
name := "SparkStreaming"
version := "1.0"
scalaVersion := "2.11.7"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % "1.2.1","org.apache.spark" % "spark-mllib_2.10" % "1.2.1")
scalacOptions += "-deprecation"

compilation error :

[root@hadoop1 SparkStreaming]# sbt compile
[info] Set current project to SparkStreaming (in build file:/root/SparkStreaming/)
[info] Compiling 2 Scala sources to /root/SparkStreaming/target/scala-2.11/classes...
[error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:7: object twitter is not a member of package org.apache.spark.streaming
[error] import org.apache.spark.streaming.twitter._
[error]                                   ^
[error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:29: not found: value TwitterUtils
[error]     val tweets = TwitterUtils.createStream(ssc, None)
[error]                  ^
[error] two errors found
[error] (compile:compile) Compilation failed
[error] Total time: 3 s, completed Sep 15, 2016 12:57:15 PM

thanks

Re: sbt compilation error

aliyesami — Fri, 16 Sep 2016 00:07:27 GMT

the twitter related jars I have placed in the 'resource' folder as shown below :

[root@hadoop1 resources]# pwd /root/SparkStreaming/src/main/resources [root@hadoop1 resources]# ls dstream-twitter_2.11-0.1.0-SNAPSHOT.jar target twitter4j-core-4.0.4.ziptwitter4j-core-4.0.4.jar twitter4j-stream-4.0.4.jar [root@hadoop1 resources]# jar tvf dstream-twitter_2.11-0.1.0-SNAPSHOT.jar | grep TwitterUtils 5862 Wed Aug 10 11:11:36 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils$.class 4749 Wed Aug 10 11:11:34 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils.class [root@hadoop1 resources]#

Re: sbt compilation error

aliyesami — Fri, 16 Sep 2016 02:51:37 GMT

this build.sbt fix the issue and now it compiles the package fine

[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

[root@hadoop1 TwitterPopularTags]#