Hey Divya, There are a couple of ways to do this. The main flow is this however.
Load/parse the data into dataframes. It seems like you have already done this but since you didn't pass along that snippet I'm just going to make something up. You did mention you were using the spark-csv package so the example is doing the same.
val sqlContext = new SQLContext(sc)
val df = sqlContext.read
.format("com.databricks.spark.csv")
.option("header", "true") // Use first line of all files as header
.option("inferSchema", "true") // Automatically infer data types
.load("cars.csv");
Write the dataframe data to the HDFS location where you plan to create the Hive external table or the directory for an existing Hive table. df.select("year", "model").write()
.format("com.databricks.spark.csv")
.option("header", "true")
.save("hdfs://hdfs_location/newcars.csv");
Create the external Hive table by creating a HiveSQLContext val hiveSQLContext = new org.apache.spark.sql.hive.HiveContext(sc)
//Several other options can be passed in here for other formats, partitions, etc
hiveSQLContext.sql("CREATE EXTERNAL TABLE cars(year INT, model STRING) STORED AS TEXTFILE LOCATION 'hdfs_location'");
Query the Hive table with whatever query you wish // Queries are expressed in HiveQL
hiveSQLContext.sql("SELECT * FROM cars").collect().foreach(println)
... View more
