@sandeep agarwal See this post about Spark vs Tez: https://community.hortonworks.com/questions/5408/spark-vs-tez.html#comment-6248 ... View more

Great article Wes! ... View more

@Vitor Batista How difficult is it for you to upgrade spark? When you run spark on yarn (with hortonworks), the upgrade process is really simple, like the steps describe here: http://hortonworks.com/hadoop-tutorial/apache-spark-1-5-1-technical-preview-with-hdp-2-3/ This is one of the advantages to run spark on yarn instead of spark standalone mode. Have you considered this option as well? ... View more

@Kit Menke If you want to access your table from hive, you have two options: 1- create table ahead and use df.write.fromat("orc") 2- use Brandon's suggestion here, register df as temp_table and do create table as select from temp_table. See code examples here: https://community.hortonworks.com/questions/6023/orgapachesparksparkexception-task-failed-while-wri.html#answer-6048 If you use saveAsTable function, it will create a table in hive metastore, but hive wont be able to query it. Only spark can use the table with this method. ... View more

Awesome! ... View more

Additionally, if you want to change number of partitions (and then parallelism) of an existing RDD, you can use rdd.repartition(8) See the comments and tests from here: https://community.hortonworks.com/questions/5825/best-way-to-select-distinct-values-from-multiple-c.html ... View more