zeppelin twitter StreamingContext...not found

[#1]

Hi All,

while trying the twitter-streaming zeppelin notebook example in hdp 25., I keep getting this error

<console>:44: error: not found: type StreamingContext

Please help.

import org.apache.spark.streaming._

import org.apache.spark.streaming.twitter._

import org.apache.spark.storage.StorageLevel

import scala.io.Source

import scala.collection.mutable.HashMap

import java.io.File import org.apache.log4j.Logger

import org.apache.log4j.Level

<console>:44: error: not found: type StreamingContext

val ssc = new StreamingContext(sc, Seconds(2)) ^

<console>:45: error: not found: value TwitterUtils val tweets = TwitterUtils.createStream(ssc, None)

Thanks,

Avijeet

[#2]

@Avijeet Dash

Try loading https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-twitter_2.11/1.5.2 under spark interpreter as a dependency

[#3]

hi @pankaj singh

I am trying to use the local repo @ /usr/hdp/current/zeppelin-server/local-repo/2BXQ86S9Q

Which has the jar files

Also isn't StreamContext part of Spark core libraries?

Why does it complain StremContext cont found.

Thanks,

Avijeet

[#4]

@Avijeet Dash I was trying some use-case and was facing similar issue. When i added the library the issue got resolved. So thought of sharing with you

[#5]

thanks @pankaj singh, how did you add the repository to zeppelin?

[#6]

@Avijeet Dash

Have a look at https://zeppelin.apache.org/docs/latest/manual/dependencymanagement.html

[#7]

Hi All,

I tried this again with a new zeppelin instance - I can see the Jars getting added in log

still I get the same error

<console>:44: error: not found: type StreamingContext val ssc = new StreamingContext(sc, Seconds(2))

Logs (I have masked the ip address)

--------

a-support-3.0.3.jar with timestamp 1488442845223 INFO [2017-03-02 08:20:45,225] ({pool-2-thread-3} SparkInterpreter.java[open]:684) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-media-support/3.0.3/twitter4j-media-support-3.0.3.jar) INFO [2017-03-02 08:20:45,227] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-stream/3.0.3/twitter4j-stream-3.0.3.jar at http://000.000.0.00:43221/jars/twitter4j-stream-3.0.3.jar with timestamp 1488442845227 INFO [2017-03-02 08:20:45,228] ({pool-2-thread-3} SparkInterpreter.java[open]:684) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-stream/3.0.3/twitter4j-stream-3.0.3.jar) INFO [2017-03-02 08:20:45,231] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-core/3.0.3/twitter4j-core-3.0.3.jar at http://000.000.0.00:43221/jars/twitter4j-core-3.0.3.jar with timestamp 1488442845231 INFO [2017-03-02 08:20:45,235] ({pool-2-thread-3} SparkInterpreter.java[open]:684) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-core/3.0.3/twitter4j-core-3.0.3.jar) INFO [2017-03-02 08:20:45,237] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/unused-1.0.0.jar at http://000.000.0.00:43221/jars/unused-1.0.0.jar with timestamp 1488442845237 INFO [2017-03-02 08:20:45,238] ({pool-2-thread-3} SparkInterpreter.java[open]:703) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/unused-1.0.0.jar) INFO [2017-03-02 08:20:45,307] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/twitter4j-core-3.0.3.jar at http://000.000.0.00:43221/jars/twitter4j-core-3.0.3.jar with timestamp 1488442845307 INFO [2017-03-02 08:20:45,313] ({pool-2-thread-3} SparkInterpreter.java[open]:703) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/twitter4j-core-3.0.3.jar) INFO [2017-03-02 08:20:45,314] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/spark-streaming-twitter_2.11-1.5.2.jar at http://000.000.0.00:43221/jars/spark-streaming-twitter_2.11-1.5.2.jar with timestamp 1488442845314 INFO [2017-03-02 08:20:45,314] ({pool-2-thread-3} SparkInterpreter.java[open]:703) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/spark-streaming-twitter_2.11-1.5.2.jar) INFO [2017-03-02 08:20:45,314] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/twitter4j-stream-3.0.3.jar at http://000.000.0.00:43221/jars/twitter4j-stream-3.0.3.jar with timestamp 1488442845314 INFO [2017-03-02 08:20:45,315] ({pool-2-thread-3} SparkInterpreter.java[open]:703) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/twitter4j-stream-3.0.3.jar) INFO [2017-03-02 08:20:50,921] ({pool-2-thread-3} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1488442776622 finished by scheduler org.apache.zeppelin.spark.SparkInterpreter776184430 INFO [2017-03-02 08:24:23,965] ({pool-2-thread-5} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1488443063964 started by scheduler org.apache.zeppelin.spark.SparkInterpreter776184430 INFO [2017-03-02 08:24:28,131] ({pool-2-thread-5} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1488443063964 finished by scheduler org.apache.zeppelin.spark.SparkInterpreter776184430

[#8 · Accepted solution]

StreamingContext expects spark Conf. Can you pass conf instead of sc context?

A StreamingContext object can be created from a SparkConf object. From https://spark.apache.org/docs/latest/streaming-programming-guide.html#initializing-streamingcontext

import org.apache.spark._
import org.apache.spark.streaming._

val conf = new SparkConf().setAppName(appName).setMaster(master)
val ssc = new StreamingContext(conf, Seconds(1))

[#9]

Hi @Artem Ervits

I am getting below error now - which class is it complaining about

SparkConf OR StremingContext OR TwitterUtils ?

import org.apache.spark._ import org.apache.spark.streaming._ import org.apache.spark.streaming.twitter._ import org.apache.spark.storage.StorageLevel import scala.io.Source import scala.collection.mutable.HashMap import java.io.File import org.apache.log4j.Logger import org.apache.log4j.Level import sys.process.stringSeqToProcess <console>:40: error: not found: type SparkConf val conf = new SparkConf().setAppName("avi_twitter").setMaster("local") ^ <console>:41: error: not found: type StreamingContext val ssc = new StreamingContext(conf, Seconds(2)) ^ <console>:42: error: not found: value TwitterUtils val tweets = TwitterUtils.createStream(ssc, None, Array("#hadoop", "#bigdata", "#spark", "#hortonworks", "#HDP"))

[#10]

Hi,

It worked after I added imports again in the script.

Thanks,

Avijeet