Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

getText method is not available while working with Spark Twitter streaming

Solved Go to solution

getText method is not available while working with Spark Twitter streaming

Contributor

Im working on a spark streaming program on Twitter data. In my code,

object TwitterDataAnalysis {
  def main(args: Array[String]) {
    if(args.length < 4) {
      System.err.println("Usage: TwitterPopularTags <consumer key> <consumer secret> " +
                          "<access token> <access token secret> [<filters>]")
      System.exit(1)
    }
    val Array(consumerKey, consumerSecret, accessToken, accessTokenSecret) = args.take(4)
    val filters = args.takeRight(args.length - 4)    
    System.setProperty("twitter4j.oauth.consumerKey", consumerKey)
    System.setProperty("twitter4j.oauth.consumerSecret", consumerSecret)
    System.setProperty("twitter4j.oauth.accessToken", accessToken)
    System.setProperty("twitter4j.oauth.accessTokenSecret", accessTokenSecret)    
    val sparkConf = new SparkConf().setAppName("Twitter's trending HashTags").setMaster("local[2]")
    val ssc       = new StreamingContext(sparkConf,Seconds(5))
    val stream    = TwitterUtils.createStream(ssc, None, filters)
    val tweets    = stream.flatMap(tweet => tweet.getText.split(" ")).filter(_.startsWith("#"))
  }
}

 

 

I haven't completed the program as Im getting an error with the line:

 

 val tweets = stream.flatMap(tweet => tweet.getText.split(" ")).filter(_.startsWith("#"))

It says:

 

 

value getText is not a member of type parameter T

 

These are the dependencies in my pom.xml file.

 <!-- Spark Dependency -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.6.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-stream -->
<dependency>
    <groupId>org.twitter4j</groupId>
    <artifactId>twitter4j-stream</artifactId>
    <version>4.0.6</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-twitter_2.10 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-twitter_2.10</artifactId>
    <version>1.6.3</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.10</artifactId>
    <version>2.1.1</version>
</dependency>


My import statements:

import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.SparkContext
import org.apache.spark.streaming.twitter._
import org.apache.spark.SparkConf
import StreamingContext._

Can anyone tell what is the mistake Im doing ? I couldn't go further the bolded line as Im not getting functions after that. Is there anything I need to change in the code ?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: getText method is not available while working with Spark Twitter streaming

Expert Contributor

Hi Sidhartha,

 

It appears you are using a newer version of spark-streaming (2.1.1).  The spark streaming twitter includes spark streaming 1.6.3 and you are using spark 1.6, this may be causing conflicts.  There is no need to include the spark-streaming dependency as it will be pulled in with the spark streaming twitter as a transitive dependency.

 

Jason

3 REPLIES 3

Re: getText method is not available while working with Spark Twitter streaming

Expert Contributor

Hi Sidhartha,

 

It appears you are using a newer version of spark-streaming (2.1.1).  The spark streaming twitter includes spark streaming 1.6.3 and you are using spark 1.6, this may be causing conflicts.  There is no need to include the spark-streaming dependency as it will be pulled in with the spark streaming twitter as a transitive dependency.

 

Jason

Re: getText method is not available while working with Spark Twitter streaming

Contributor

Okay. So what are the Jar files that I need to change. I meant, match the versions. Because if I remove the spark-streaming dependency, Im getting error at this line:

val ssc       = new StreamingContext(sparkConf,Seconds(5))
msg: not found: type StreamingContext

Re: getText method is not available while working with Spark Twitter streaming

Expert Contributor
You still need the spark-streaming dependency, but instead of version 2.1.1 you will want to match your spark core version of 1.6.3.