This looks strange. Your console output listed the below lines     com.databricks#spark-avro_2.10 added as a dependency org.apache.avro#avro-mapred added as a dependency   Can you try once with :     --packages com.databricks:spark-avro_2.10:1.0.0,org.apache.avro:avro-mapred:1.6.3   I can sense some version compatibility issues of avro-mapred with spark-avro. ... View more

Try starting spark-shell with following packages:   --packages com.databricks:spark-avro_2.10:2.0.1,org.apache.avro:avro-mapred:1.7.7   ... View more

What Python version you are using.    You may want to refer : http://www.cloudera.com/documentation/enterprise/5-5-x/topics/spark_ipython.html ... View more

This looks weird. And can you confirm that    http://192.168.88.28:55310/jars/phoenix-1.2.0-client.jar is still not present?   Spark keeps all JARs specified by --jars option in job's temp directory on each executor nodes [1]. There must be some sort of OS settings which lead the deletion of existing phoenix jar from temp and when Spark Context is unable to find it at its usual location it tries to download it from the given location. However this should not happen until the temp directory is actively accessed by the job or process.   You can try bundling that JAR with your Spark JAR and then refer it in spark-submit.  I suspect, you will need again 20 odd days to test this workaround 🙂 ... View more

Add below dependency as well:   groupId = org.apache.spark artifactId = spark-streaming-flume_2.10 version = 1.6.1   See here for pull based configuration.  ... View more

Yes, YARN provides this flexibility. Here you can find the detailed answer.    For CDH there is a "Spark" service, which meant for YARN and another is "Spark Standalone" service which runs it's daemons standalone on the specified nodes.    YARN will do the work for you if you want to test the multiple versions simultaneously. You should have your multiple versions on Gateway Host and then you can launch Spark applications from there. ... View more