Great sample code. In most of my Spark apps when working with Parquet, I have a few configurations that help. There are some SparkConfigurations that will help working with Parquet files. val sparkConf = new SparkConf()
sparkConf.set("spark.sql.parquet.compression.codec", "snappy")
sparkConf.set("spark.sql.parquet.mergeSchema", "true")
sparkConf.set("spark.sql.parquet.binaryAsString", "true")
sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)
sparkConf.set("spark.sql.tungsten.enabled", "true")
sparkConf.set("spark.eventLog.enabled", "true")
sparkConf.set("spark.io.compression.codec", "snappy")
sparkConf.set("spark.rdd.compress", "true")
sparkConf.set("spark.streaming.backpressure.enabled", "true")
Some of the compression items are really important for different use cases. You also often need to turn them off or switch which codec you use depending on use case (batch, streaming, sql, large, small, many partitions, ...) EventLog enabled so you can look at how those parquet files are worked with in DAGs and metrics. Before you right some SparkSQL on that file, make sure you register a table name. If you don't want to do a write that will file if the directory/file already exists, you can choose Append mode to add to it. It depends on your use case. df1.registerTempTable("MyTableName")
val results = sqlContext.sql("SELECT name FROM MyTableName")
df1.write.format("parquet").mode(org.apache.spark.sql.SaveMode.Append).parquet("data.parquet") If you want to look at the data from the command line after you write it, you can download parquet tools. This requires the Java JDK, git and Maven installed. git clone -b apache-parquet-1.8.0 https://github.com/apache/parquet-mr.git
cd parquet-mr
cd parquet-tools
mvn clean package -Plocal
... View more
