@Jan J I would store in HDFS. I would leverage Hive...See this http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables When you use spark to read hive tables then you are using spark features to read data from hive. In terms of performance, we are not really testing spark sql performance, but hiveql instead? You are testing performance of SparkSql feature with Hive All the answers are in the overview http://spark.apache.org/docs/latest/sql-programming-guide.html#overview SQL One use of Spark SQL is to execute SQL queries written using either a basic SQL syntax or HiveQL. Spark SQL can also be used to read data from an existing Hive installation. For more on how to configure this feature, please refer to the Hive Tables section. When running SQL from within another programming language the results will be returned as a DataFrame. You can also interact with the SQL interface using thecommand-line or over JDBC/ODBC. DataFrames A DataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood. DataFrames can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing RDDs. The DataFrame API is available in Scala, Java, Python, and R. Datasets A Dataset is a new experimental interface added in Spark 1.6 that tries to provide the benefits of RDDs (strong typing, ability to use powerful lambda functions) with the benefits of Spark SQL’s optimized execution engine. A Dataset can be constructed from JVM objects and then manipulated using functional transformations (map, flatMap, filter, etc.). The unified Dataset API can be used both in Scala and Java. Python does not yet have support for the Dataset API, but due to its dynamic nature many of the benefits are already available (i.e. you can access the field of a row by name naturally row.columnName ). Full python support will be added in a future release.
... View more
