Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 09-15-2016 05:05 PM
utilitiesscala.txtprinttweetsscala.txtmy build file is ,I am attaching the source code .
[root@hadoop1 SparkStreaming]# more build.sbt
name := "SparkStreaming"
version := "1.0"
scalaVersion := "2.11.7"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % "1.2.1","org.apache.spark" % "spark-mllib_2.10" % "1.2.1")
scalacOptions += "-deprecation"
compilation error :
[root@hadoop1 SparkStreaming]# sbt compile [info] Set current project to SparkStreaming (in build file:/root/SparkStreaming/) [info] Compiling 2 Scala sources to /root/SparkStreaming/target/scala-2.11/classes... [error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:7: object twitter is not a member of package org.apache.spark.streaming [error] import org.apache.spark.streaming.twitter._ [error] ^ [error] /root/SparkStreaming/src/main/scala/PrintTweets.scala:29: not found: value TwitterUtils [error] val tweets = TwitterUtils.createStream(ssc, None) [error] ^ [error] two errors found [error] (compile:compile) Compilation failed [error] Total time: 3 s, completed Sep 15, 2016 12:57:15 PM
thanks
Created 09-15-2016 07:51 PM
this build.sbt fix the issue and now it compiles the package fine
[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
[root@hadoop1 TwitterPopularTags]#
Created 09-15-2016 05:07 PM
the twitter related jars I have placed in the 'resource' folder as shown below :
[root@hadoop1 resources]# pwd /root/SparkStreaming/src/main/resources [root@hadoop1 resources]# ls dstream-twitter_2.11-0.1.0-SNAPSHOT.jar target twitter4j-core-4.0.4.ziptwitter4j-core-4.0.4.jar twitter4j-stream-4.0.4.jar [root@hadoop1 resources]# jar tvf dstream-twitter_2.11-0.1.0-SNAPSHOT.jar | grep TwitterUtils 5862 Wed Aug 10 11:11:36 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils$.class 4749 Wed Aug 10 11:11:34 EDT 2016 org/apache/spark/streaming/twitter/TwitterUtils.class [root@hadoop1 resources]#
Created 09-15-2016 07:51 PM
this build.sbt fix the issue and now it compiles the package fine
[root@hadoop1 TwitterPopularTags]# more build.sbt name := "TwitterPopularTags"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
[root@hadoop1 TwitterPopularTags]#