question Re: Spark 1.6 Technical Preview in Archives of Support Questions (Read Only)

Spark 1.6 Technical Preview

vshukla — Tue, 05 Jan 2016 03:12:27 GMT

Team,

Congratulations. We just published Spark 1.6 technical preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

Please take it out for a spin.

Re: Spark 1.6 Technical Preview

aervits — Tue, 05 Jan 2016 03:13:44 GMT

+1 for including DataSet API example, incredible work team!

Re: Spark 1.6 Technical Preview

aervits — Tue, 05 Jan 2016 04:56:59 GMT

@vshukla running through the tutorial, in the step 1 of

Accessing the Spark SQL Thrift Server for JDBC/ODBC Access

sudo root yields

"sudo: root: command not found"

Re: Spark 1.6 Technical Preview

aervits — Tue, 05 Jan 2016 05:01:18 GMT

in Launch SparkR: step

it should be

./bin/sparkR

Re: Spark 1.6 Technical Preview

aervits — Tue, 05 Jan 2016 05:04:13 GMT

in using DataSet API step 1, it should be

su spark 

./bin/spark-shell --num-executors 2--executor-memory 512m--master yarn-client

and not

su spark ./bin/spark-shell --num-executors 2--executor-memory 512m--master yarn-client

otherwise it doesn't work

Re: Spark 1.6 Technical Preview

vshukla — Tue, 05 Jan 2016 05:26:05 GMT

Thanks @Artem Ervits. Typos fixed.

Re: Spark 1.6 Technical Preview

stevel — Tue, 05 Jan 2016 23:36:20 GMT

1. The clickalbe link is: apache-spark-1-6-technical-preview-with-hdp-2-3

2, @vshukla: is there a tarball somewhere too? That's useful client-side

Re: Spark 1.6 Technical Preview

phargis — Fri, 05 Feb 2016 04:03:36 GMT

This tutorial skipped one set of instructions to eliminate use of Yarn HistoryServer. These are the required steps:

Ensure “spark-defaults.conf” doesn’t have any YARN history service related properties enabled.

If this tech preview is installed on a node where Spark was already there, there may be Spark properties set related to YARN ATS. Make sure you have disabled the following properties in your “spark-defaults.conf” file by adding a '#' in front of each setting.

#spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider

#spark.history.ui.port 18080

#spark.yarn.historyServer.address sandbox.hortonworks.com:18080

#spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService

Re: Spark 1.6 Technical Preview

rhryniewicz — Thu, 03 Mar 2016 03:19:08 GMT

Thanks. Incorporated feedback to A Lap Around Apache Spark - http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

Re: Spark 1.6 Technical Preview

rhryniewicz — Thu, 03 Mar 2016 03:20:39 GMT

Thanks. Incorporated feedback to A Lap Around Apache Spark - http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

Re: Spark 1.6 Technical Preview

rhryniewicz — Thu, 03 Mar 2016 03:25:16 GMT

Could not find this in Spark 1.6 Tech Preview. The latest updates to the "Accessing the Spark SQL Thrift Server for JDBC/ODBC Access" have been updated, tested, and incorporated into http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

Re: Spark 1.6 Technical Preview

aervits — Thu, 03 Mar 2016 03:25:54 GMT

@Robert Hryniewicz yes I believe Vinay fixed it when I reported it.