Please check if numpy is actually installed on all of the nodemanagers, if not, install it using below command (for python2.x) :   pip install numpy   If already installed, let us know the following:    1) Can you execute the same command outside of hue i.e. using Spark2-submit ? Mention the full command here. 2) What spark command you use in Hue? ... View more

The valuable information is at very bottom:  NameError: name 'master' is not defined   Please make sure you have defined variable "master" in your code. Or if you are specifying master via spark-submit, you should not set it in code. ... View more

You don't need to export as a JAR for unit testing. You can do :   SparkConf().setMaster(local[2]) and run the program as usual Java application in IDE.  Also make sure that you have all the dependent libraries in the classpath. ... View more

What Python version you are using.    You may want to refer : http://www.cloudera.com/documentation/enterprise/5-5-x/topics/spark_ipython.html ... View more

This looks weird. And can you confirm that    http://192.168.88.28:55310/jars/phoenix-1.2.0-client.jar is still not present?   Spark keeps all JARs specified by --jars option in job's temp directory on each executor nodes [1]. There must be some sort of OS settings which lead the deletion of existing phoenix jar from temp and when Spark Context is unable to find it at its usual location it tries to download it from the given location. However this should not happen until the temp directory is actively accessed by the job or process.   You can try bundling that JAR with your Spark JAR and then refer it in spark-submit.  I suspect, you will need again 20 odd days to test this workaround 🙂 ... View more

Add below dependency as well:   groupId = org.apache.spark artifactId = spark-streaming-flume_2.10 version = 1.6.1   See here for pull based configuration.  ... View more

Yes, YARN provides this flexibility. Here you can find the detailed answer.    For CDH there is a "Spark" service, which meant for YARN and another is "Spark Standalone" service which runs it's daemons standalone on the specified nodes.    YARN will do the work for you if you want to test the multiple versions simultaneously. You should have your multiple versions on Gateway Host and then you can launch Spark applications from there. ... View more

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.util.ThreadUtils$.runInNewThread$default$2()Z    Compare your code with below line:   .setMaster("local[2]")   BTW which version of Spark Streaming you are usning? ... View more

You need to assign number of threads to spark while running master on local, most obvious choice is 2, 1 to recieve the data and 1 to process them.   so the correct code should be : .setMaster("local[2]")     If your file is not too big change to : val ssc = new StreamingContext(sc, Seconds(1))   You have stopped the streaming but forgot to start it:   file.foreachRDD(t=> { val test = t.map(x => (x.split(" ")(0)+";"+x.split(" ")(1), 1)).reduceByKey((x,y) => x+y) test.saveAsTextFile("/root/file/file1") }) sc.start() sc.awaitTermination()   As of now dont use sc.stop()     ... View more

Seems that there is some glitch in your code. It would be much easy if you could post your code. ... View more

From your code :   val textFile = sc.textFileStream( "/root/file/test" ) textFile.foreachRDD(t=> { val test = t.map(x => (x.split( " " )( 0 )+ ";" +x.split( " " )( 1 ) , 1 )).reduceByKey((x , y) => x+y) test.saveAsTextFile( "/root/file/file1" ) ; })   Mind the t.map( ) not file.map( ) ... View more

You have a handy method bundled with Spark "foreachRDD":    val file = ssc.textFileStream("/root/file/test") file.foreachRDD(t=> { var test=t.map() //DO the map stuff here test.saveAsTextFiles("/root/file/file1") }) sc.stop()       ... View more