Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark 1.6 Technical Preview

avatar

Team,

Congratulations. We just published Spark 1.6 technical preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

Please take it out for a spin.

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
11 REPLIES 11

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar

This tutorial skipped one set of instructions to eliminate use of Yarn HistoryServer. These are the required steps:

Ensure “spark-defaults.conf” doesn’t have any YARN history service related properties enabled.

If this tech preview is installed on a node where Spark was already there, there may be Spark properties set related to YARN ATS. Make sure you have disabled the following properties in your “spark-defaults.conf” file by adding a '#' in front of each setting.

#spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider

#spark.history.ui.port 18080

#spark.yarn.historyServer.address sandbox.hortonworks.com:18080

#spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService