Member since
09-29-2015
122
Posts
159
Kudos Received
26
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7065 | 11-12-2016 12:32 AM | |
2078 | 10-05-2016 08:08 PM | |
2872 | 08-02-2016 11:29 PM | |
24014 | 06-24-2016 11:46 PM | |
2222 | 05-25-2016 11:12 PM |
04-11-2016
06:14 PM
1 Kudo
We support MR or Tez as the execution engine for Hive. We also support SparkSQL.
... View more
04-08-2016
06:22 PM
You can also switch to yarn-client mode to see more logs printed directly onto the console. Remember to switch back to yarn-cluster mode after you are done debugging.
... View more
03-24-2016
06:03 PM
1 Kudo
As Simon mentioned RDDs don't have schema attached. DataFrame (conceptually similar to a DB Table) do have an attached schema (column name, column type etc) and you can quite easily filter on a column . You can also create a DF from RDD and then go about filter. See http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/ section about programmatically specifying schema (that attaches schema to RDD to get a DataFrame) and see the section Additional DataFrame API Example to see a DF filter example.
... View more
02-17-2016
06:53 PM
1 Kudo
Officially upgrade from Spark 1.5.2 > 1.6 is part of HDP 2.4 that is coming out March 1st. The Spark 1.6 tutorial is for evaluating Spark 1.6 and not for evaluation of upgrade instructions.
... View more
02-03-2016
02:27 AM
2 Kudos
Can you please try using HiveContext and report back? @Maleeha Qazi
... View more
02-03-2016
01:47 AM
2 Kudos
Think this may be related to https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3CCAAswR-5=az1SPxo8EaQvOs2JMh=V82zMfAz67PqGy+CQqrrc=Q@mail.gmail.com%3E What is your spark-shell mode? Yarn-cluster or yarn-client?
... View more
01-29-2016
07:59 PM
1 Kudo
Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview you can get it to work with Spark 1.5.x in HDP 2.3.4. In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)
... View more
01-29-2016
07:56 PM
1 Kudo
Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4. In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)
... View more
01-26-2016
08:30 PM
1 Kudo
We don't provide Spark on Windows anymore. The last Spark on Windows was 1.3.1 with HDP 2.2.x
... View more