About vshukla

vshukla · ‎04-19-2016

See Slides & video for the session.

vshukla · ‎04-11-2016

We support MR or Tez as the execution engine for Hive. We also support SparkSQL.

vshukla · ‎04-08-2016

You can also switch to yarn-client mode to see more logs printed directly onto the console. Remember to switch back to yarn-cluster mode after you are done debugging.

vshukla · ‎03-24-2016

As Simon mentioned RDDs don't have schema attached. DataFrame (conceptually similar to a DB Table) do have an attached schema (column name, column type etc) and you can quite easily filter on a column . You can also create a DF from RDD and then go about filter. See http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/ section about programmatically specifying schema (that attaches schema to RDD to get a DataFrame) and see the section Additional DataFrame API Example to see a DF filter example.

vshukla · ‎02-17-2016

Officially upgrade from Spark 1.5.2 > 1.6 is part of HDP 2.4 that is coming out March 1st. The Spark 1.6 tutorial is for evaluating Spark 1.6 and not for evaluation of upgrade instructions.

vshukla · ‎02-03-2016

Can you please try using HiveContext and report back? @Maleeha Qazi

vshukla · ‎02-03-2016

Think this may be related to https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3CCAAswR-5=az1SPxo8EaQvOs2JMh=V82zMfAz67PqGy+CQqrrc=Q@mail.gmail.com%3E What is your spark-shell mode? Yarn-cluster or yarn-client?

vshukla · ‎01-29-2016

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview you can get it to work with Spark 1.5.x in HDP 2.3.4. In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

vshukla · ‎01-29-2016

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4. In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

vshukla · ‎01-26-2016

We don't provide Spark on Windows anymore. The last Spark on Windows was 1.3.1 with HDP 2.2.x

Online	Offline
Last Visited	‎12-08-2017 06:12 PM

Member Since	‎09-29-2015 03:27 AM
Last Visited	‎12-08-2017 06:12 PM
Posts	122
Kudos received	149

Cloudera Community

Re: Exception: Session not found, Livy server woul...

Re: Spark 2 and Zeppelin

Re: Does the HiveContext object expire in Zeppelin...

Re: Adding libraries to Zeppelin

Re: Zeppelin Error - bash interpreter running a Hi...

Running Spark in Production?

Re: Unable to use hive.execution.engine=spark In H...

Re: Spark job submit log messages on console

Re: How to filter a Spark RDD based on particular ...

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

Re: Spark 1.3.1 not pulling all hive metadata when...

Re: HIVE / SparkSQL '.parquet not a SequenceFile '

Re: Spark 1.6 with Zeppelin

Re: Spark 1.5 with Zeppelin - but sc.version print...

Re: spark on hdp 2.3.4 for Windows