Spark 1.6 Technical Preview

[#1]

Team,

Congratulations. We just published Spark 1.6 technical preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

Please take it out for a spin.

[#2 · Accepted solution]

1. The clickalbe link is: apache-spark-1-6-technical-preview-with-hdp-2-3

2, @vshukla: is there a tarball somewhere too? That's useful client-side

[#3]

This tutorial skipped one set of instructions to eliminate use of Yarn HistoryServer. These are the required steps:

Ensure “spark-defaults.conf” doesn’t have any YARN history service related properties enabled.

If this tech preview is installed on a node where Spark was already there, there may be Spark properties set related to YARN ATS. Make sure you have disabled the following properties in your “spark-defaults.conf” file by adding a '#' in front of each setting.

#spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider

#spark.history.ui.port 18080

#spark.yarn.historyServer.address sandbox.hortonworks.com:18080

#spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService