getText method is not available while working with Spark Twitter streaming

[#1]

Im working on a spark streaming program on Twitter data. In my code,

object TwitterDataAnalysis {
  def main(args: Array[String]) {
    if(args.length < 4) {
      System.err.println("Usage: TwitterPopularTags <consumer key> <consumer secret> " +
                          "<access token> <access token secret> [<filters>]")
      System.exit(1)
    }
    val Array(consumerKey, consumerSecret, accessToken, accessTokenSecret) = args.take(4)
    val filters = args.takeRight(args.length - 4)    
    System.setProperty("twitter4j.oauth.consumerKey", consumerKey)
    System.setProperty("twitter4j.oauth.consumerSecret", consumerSecret)
    System.setProperty("twitter4j.oauth.accessToken", accessToken)
    System.setProperty("twitter4j.oauth.accessTokenSecret", accessTokenSecret)    
    val sparkConf = new SparkConf().setAppName("Twitter's trending HashTags").setMaster("local[2]")
    val ssc       = new StreamingContext(sparkConf,Seconds(5))
    val stream    = TwitterUtils.createStream(ssc, None, filters)
    val tweets    = stream.flatMap(tweet => tweet.getText.split(" ")).filter(_.startsWith("#"))
  }
}

I haven't completed the program as Im getting an error with the line:

 val tweets = stream.flatMap(tweet => tweet.getText.split(" ")).filter(_.startsWith("#"))

It says:

value getText is not a member of type parameter T

These are the dependencies in my pom.xml file.

 <!-- Spark Dependency -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.6.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-stream -->
<dependency>
    <groupId>org.twitter4j</groupId>
    <artifactId>twitter4j-stream</artifactId>
    <version>4.0.6</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-twitter_2.10 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-twitter_2.10</artifactId>
    <version>1.6.3</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.10</artifactId>
    <version>2.1.1</version>
</dependency>

My import statements:

import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.SparkContext
import org.apache.spark.streaming.twitter._
import org.apache.spark.SparkConf
import StreamingContext._

Can anyone tell what is the mistake Im doing ? I couldn't go further the bolded line as Im not getting functions after that. Is there anything I need to change in the code ?

[#2 · Accepted solution]

Hi Sidhartha,

It appears you are using a newer version of spark-streaming (2.1.1).  The spark streaming twitter includes spark streaming 1.6.3 and you are using spark 1.6, this may be causing conflicts.  There is no need to include the spark-streaming dependency as it will be pulled in with the spark streaming twitter as a transitive dependency.

Jason

[#3]

Okay. So what are the Jar files that I need to change. I meant, match the versions. Because if I remove the spark-streaming dependency, Im getting error at this line:

val ssc       = new StreamingContext(sparkConf,Seconds(5))
msg: not found: type StreamingContext

[#4]

You still need the spark-streaming dependency, but instead of version 2.1.1 you will want to match your spark core version of 1.6.3.