Created 09-15-2016 05:05 PM
utilitiesscala.txtprinttweetsscala.txtmy build file is ,I am attaching the source code .
[root@hadoop1 SparkStreaming]# more build.sbt name := "SparkStreaming" version := "1.0" scalaVersion := "2.11.7" resolvers += "Akka Repository" at "http://repo.akka.io/releases/" libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % "1.2.1","org.apache.spark" % "spark-mllib_2.10" % "1.2.1") scalacOptions += "-deprecation"
compilation error :
[root@hadoop1 SparkStreaming]# sbt compile [info] Set current project to SparkStreaming (in build file:/root/SparkStreaming/) [info] Compiling 2 Scala sources to /root/SparkStreaming/target/scala-2.11/classes... [error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:7: object twitter is not a member of package org.apache.spark.streaming [error] import org.apache.spark.streaming.twitter._ [error] ^ [error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:29: not found: value TwitterUtils [error] val tweets = TwitterUtils.createStream(ssc, None) [error] ^ [error] two errors found [error] (compile:compile) Compilation failed [error] Total time: 3 s, completed Sep 15, 2016 12:57:15 PM
thanks
Created 09-15-2016 07:51 PM
this build.sbt fix the issue and now it compiles the package fine
[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
[root@hadoop1 TwitterPopularTags]#
Created 09-15-2016 05:07 PM
the twitter related jars I have placed in the 'resource' folder as shown below :
[root@hadoop1 resources]# pwd /root/SparkStreaming/src/main/resources [root@hadoop1 resources]# ls dstream-twitter_2.11-0.1.0-SNAPSHOT.jar target twitter4j-core-4.0.4.ziptwitter4j-core-4.0.4.jar twitter4j-stream-4.0.4.jar [root@hadoop1 resources]# jar tvf dstream-twitter_2.11-0.1.0-SNAPSHOT.jar | grep TwitterUtils 5862 Wed Aug 10 11:11:36 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils$.class 4749 Wed Aug 10 11:11:34 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils.class [root@hadoop1 resources]#
Created 09-15-2016 07:51 PM
this build.sbt fix the issue and now it compiles the package fine
[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
[root@hadoop1 TwitterPopularTags]#