I had the same issue, increasing the heap space for HDFS helped. Also apparently a reboot was required to disable selinux - as the command sestatus was showing enabled. ... View more

Hi @Tom McCuch Great article, I have one question - when we say "ARCHIVE, which has high storage density (petabyte of storage) but little compute power" Is its a different kind of HDD - can you please give examples how it is different from the regular HDD. Thanks, Avijeet ... View more

Hi, It worked after I added imports again in the script. Thanks, Avijeet ... View more

Hi @Artem Ervits I am getting below error now - which class is it complaining about SparkConf OR StremingContext OR TwitterUtils ? import org.apache.spark._ import org.apache.spark.streaming._ import org.apache.spark.streaming.twitter._ import org.apache.spark.storage.StorageLevel import scala.io.Source import scala.collection.mutable.HashMap import java.io.File import org.apache.log4j.Logger import org.apache.log4j.Level import sys.process.stringSeqToProcess <console>:40: error: not found: type SparkConf val conf = new SparkConf().setAppName("avi_twitter").setMaster("local") ^ <console>:41: error: not found: type StreamingContext val ssc = new StreamingContext(conf, Seconds(2)) ^ <console>:42: error: not found: value TwitterUtils val tweets = TwitterUtils.createStream(ssc, None, Array("#hadoop", "#bigdata", "#spark", "#hortonworks", "#HDP")) ... View more

Hi All, I tried this again with a new zeppelin instance - I can see the Jars getting added in log still I get the same error <console>:44: error: not found: type StreamingContext val ssc = new StreamingContext(sc, Seconds(2)) Logs (I have masked the ip address) -------- a-support-3.0.3.jar with timestamp 1488442845223 INFO [2017-03-02 08:20:45,225] ({pool-2-thread-3} SparkInterpreter.java[open]:684) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-media-support/3.0.3/twitter4j-media-support-3.0.3.jar) INFO [2017-03-02 08:20:45,227] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-stream/3.0.3/twitter4j-stream-3.0.3.jar at http://000.000.0.00:43221/jars/twitter4j-stream-3.0.3.jar with timestamp 1488442845227 INFO [2017-03-02 08:20:45,228] ({pool-2-thread-3} SparkInterpreter.java[open]:684) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-stream/3.0.3/twitter4j-stream-3.0.3.jar) INFO [2017-03-02 08:20:45,231] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-core/3.0.3/twitter4j-core-3.0.3.jar at http://000.000.0.00:43221/jars/twitter4j-core-3.0.3.jar with timestamp 1488442845231 INFO [2017-03-02 08:20:45,235] ({pool-2-thread-3} SparkInterpreter.java[open]:684) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/org/twitter4j/twitter4j-core/3.0.3/twitter4j-core-3.0.3.jar) INFO [2017-03-02 08:20:45,237] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/unused-1.0.0.jar at http://000.000.0.00:43221/jars/unused-1.0.0.jar with timestamp 1488442845237 INFO [2017-03-02 08:20:45,238] ({pool-2-thread-3} SparkInterpreter.java[open]:703) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/unused-1.0.0.jar) INFO [2017-03-02 08:20:45,307] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/twitter4j-core-3.0.3.jar at http://000.000.0.00:43221/jars/twitter4j-core-3.0.3.jar with timestamp 1488442845307 INFO [2017-03-02 08:20:45,313] ({pool-2-thread-3} SparkInterpreter.java[open]:703) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/twitter4j-core-3.0.3.jar) INFO [2017-03-02 08:20:45,314] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/spark-streaming-twitter_2.11-1.5.2.jar at http://000.000.0.00:43221/jars/spark-streaming-twitter_2.11-1.5.2.jar with timestamp 1488442845314 INFO [2017-03-02 08:20:45,314] ({pool-2-thread-3} SparkInterpreter.java[open]:703) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/spark-streaming-twitter_2.11-1.5.2.jar) INFO [2017-03-02 08:20:45,314] ({pool-2-thread-3} Logging.scala[logInfo]:58) - Added JAR /usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/twitter4j-stream-3.0.3.jar at http://000.000.0.00:43221/jars/twitter4j-stream-3.0.3.jar with timestamp 1488442845314 INFO [2017-03-02 08:20:45,315] ({pool-2-thread-3} SparkInterpreter.java[open]:703) - sc.addJar(/usr/hdp/current/zeppelin-server/local-repo/2C73PQ8RJ/twitter4j-stream-3.0.3.jar) INFO [2017-03-02 08:20:50,921] ({pool-2-thread-3} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1488442776622 finished by scheduler org.apache.zeppelin.spark.SparkInterpreter776184430 INFO [2017-03-02 08:24:23,965] ({pool-2-thread-5} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1488443063964 started by scheduler org.apache.zeppelin.spark.SparkInterpreter776184430 INFO [2017-03-02 08:24:28,131] ({pool-2-thread-5} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1488443063964 finished by scheduler org.apache.zeppelin.spark.SparkInterpreter776184430 ... View more