Please check if numpy is actually installed on all of the nodemanagers, if not, install it using below command (for python2.x) :   pip install numpy   If already installed, let us know the following:    1) Can you execute the same command outside of hue i.e. using Spark2-submit ? Mention the full command here. 2) What spark command you use in Hue? ... View more

Thanks for reporting 404 for that parcel URL, apologies for the inconvenience caused.   However, I can see that the mentioned JIRA ( SPARK-22306 ) fix is present in below CDS releases :   SPARK2-2.3.0-CLOUDERA1 SPARK2-2.3.0-CLOUDERA2 SPARK2-2.3.0-CLOUDERA3 So feel free to use below link to download the CDS 2.3 release 3 parcels in the meantime:    http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera3/   ... View more

I believe you can achieve this by following the below sequence:    spark.sql("SET spark.sql.shuffle.partitions=12") Execute operations on small table spark.sql("SET spark.sql.shuffle.partitions=500") Execute operations on larger table ... View more

It would be good if you can attach full stacktrace of the error which you are seeing. As a side note, make sure that you have added Hive gateway role on the host from which you are submitting spark app.   ... View more

By  Spark 2.1 do you mean Cloudera Spark 2.0 Release 1 or Apache Spark 2.1 ?    Regarding Cloudera Spark 2.0 Release 1 or Release 2, I would like to tell that minimum required CDH version is CDH 5.7.x but you are on CDH 5.5.4.  ... View more

The valuable information is at very bottom:  NameError: name 'master' is not defined   Please make sure you have defined variable "master" in your code. Or if you are specifying master via spark-submit, you should not set it in code. ... View more

You don't need to export as a JAR for unit testing. You can do :   SparkConf().setMaster(local[2]) and run the program as usual Java application in IDE.  Also make sure that you have all the dependent libraries in the classpath. ... View more

This looks strange. Your console output listed the below lines     com.databricks#spark-avro_2.10 added as a dependency org.apache.avro#avro-mapred added as a dependency   Can you try once with :     --packages com.databricks:spark-avro_2.10:1.0.0,org.apache.avro:avro-mapred:1.6.3   I can sense some version compatibility issues of avro-mapred with spark-avro. ... View more

Try starting spark-shell with following packages:   --packages com.databricks:spark-avro_2.10:2.0.1,org.apache.avro:avro-mapred:1.7.7   ... View more

What Python version you are using.    You may want to refer : http://www.cloudera.com/documentation/enterprise/5-5-x/topics/spark_ipython.html ... View more

This looks weird. And can you confirm that    http://192.168.88.28:55310/jars/phoenix-1.2.0-client.jar is still not present?   Spark keeps all JARs specified by --jars option in job's temp directory on each executor nodes [1]. There must be some sort of OS settings which lead the deletion of existing phoenix jar from temp and when Spark Context is unable to find it at its usual location it tries to download it from the given location. However this should not happen until the temp directory is actively accessed by the job or process.   You can try bundling that JAR with your Spark JAR and then refer it in spark-submit.  I suspect, you will need again 20 odd days to test this workaround 🙂 ... View more

Add below dependency as well:   groupId = org.apache.spark artifactId = spark-streaming-flume_2.10 version = 1.6.1   See here for pull based configuration.  ... View more

Yes, YARN provides this flexibility. Here you can find the detailed answer.    For CDH there is a "Spark" service, which meant for YARN and another is "Spark Standalone" service which runs it's daemons standalone on the specified nodes.    YARN will do the work for you if you want to test the multiple versions simultaneously. You should have your multiple versions on Gateway Host and then you can launch Spark applications from there. ... View more

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.util.ThreadUtils$.runInNewThread$default$2()Z    Compare your code with below line:   .setMaster("local[2]")   BTW which version of Spark Streaming you are usning? ... View more

You need to assign number of threads to spark while running master on local, most obvious choice is 2, 1 to recieve the data and 1 to process them.   so the correct code should be : .setMaster("local[2]")     If your file is not too big change to : val ssc = new StreamingContext(sc, Seconds(1))   You have stopped the streaming but forgot to start it:   file.foreachRDD(t=> { val test = t.map(x => (x.split(" ")(0)+";"+x.split(" ")(1), 1)).reduceByKey((x,y) => x+y) test.saveAsTextFile("/root/file/file1") }) sc.start() sc.awaitTermination()   As of now dont use sc.stop()     ... View more

Seems that there is some glitch in your code. It would be much easy if you could post your code. ... View more

From your code :   val textFile = sc.textFileStream( "/root/file/test" ) textFile.foreachRDD(t=> { val test = t.map(x => (x.split( " " )( 0 )+ ";" +x.split( " " )( 1 ) , 1 )).reduceByKey((x , y) => x+y) test.saveAsTextFile( "/root/file/file1" ) ; })   Mind the t.map( ) not file.map( ) ... View more

You have a handy method bundled with Spark "foreachRDD":    val file = ssc.textFileStream("/root/file/test") file.foreachRDD(t=> { var test=t.map() //DO the map stuff here test.saveAsTextFiles("/root/file/file1") }) sc.stop()       ... View more