Support Questions

Find answers, ask questions, and share your expertise

Spark 1.6 Technical Preview

Team,

Congratulations. We just published Spark 1.6 technical preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

Please take it out for a spin.

1 ACCEPTED SOLUTION

1. The clickalbe link is: apache-spark-1-6-technical-preview-with-hdp-2-3

2, @vshukla: is there a tarball somewhere too? That's useful client-side

View solution in original post

11 REPLIES 11

Mentor

+1 for including DataSet API example, incredible work team!

Mentor

@vshukla running through the tutorial, in the step 1 of

Accessing the Spark SQL Thrift Server for JDBC/ODBC Access

sudo root yields

"sudo: root: command not found"

Could not find this in Spark 1.6 Tech Preview. The latest updates to the "Accessing the Spark SQL Thrift Server for JDBC/ODBC Access" have been updated, tested, and incorporated into http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

Mentor

@Robert Hryniewicz yes I believe Vinay fixed it when I reported it.

Mentor

in Launch SparkR: step

it should be

./bin/sparkR

Thanks. Incorporated feedback to A Lap Around Apache Spark - http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

Mentor

in using DataSet API step 1, it should be

su spark 

./bin/spark-shell --num-executors 2--executor-memory 512m--master yarn-client

and not

su spark ./bin/spark-shell --num-executors 2--executor-memory 512m--master yarn-client

otherwise it doesn't work

Thanks. Incorporated feedback to A Lap Around Apache Spark - http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

Thanks @Artem Ervits. Typos fixed.

1. The clickalbe link is: apache-spark-1-6-technical-preview-with-hdp-2-3

2, @vshukla: is there a tarball somewhere too? That's useful client-side

This tutorial skipped one set of instructions to eliminate use of Yarn HistoryServer. These are the required steps:

Ensure “spark-defaults.conf” doesn’t have any YARN history service related properties enabled.

If this tech preview is installed on a node where Spark was already there, there may be Spark properties set related to YARN ATS. Make sure you have disabled the following properties in your “spark-defaults.conf” file by adding a '#' in front of each setting.

#spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider

#spark.history.ui.port 18080

#spark.yarn.historyServer.address sandbox.hortonworks.com:18080

#spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService