Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark 1.6 Technical Preview

avatar

Team,

Congratulations. We just published Spark 1.6 technical preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

Please take it out for a spin.

1 ACCEPTED SOLUTION

avatar

1. The clickalbe link is: apache-spark-1-6-technical-preview-with-hdp-2-3

2, @vshukla: is there a tarball somewhere too? That's useful client-side

View solution in original post

11 REPLIES 11

avatar

1. The clickalbe link is: apache-spark-1-6-technical-preview-with-hdp-2-3

2, @vshukla: is there a tarball somewhere too? That's useful client-side

avatar

This tutorial skipped one set of instructions to eliminate use of Yarn HistoryServer. These are the required steps:

Ensure “spark-defaults.conf” doesn’t have any YARN history service related properties enabled.

If this tech preview is installed on a node where Spark was already there, there may be Spark properties set related to YARN ATS. Make sure you have disabled the following properties in your “spark-defaults.conf” file by adding a '#' in front of each setting.

#spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider

#spark.history.ui.port 18080

#spark.yarn.historyServer.address sandbox.hortonworks.com:18080

#spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService