1) Read Data from a file in Hadoop to a DataFrame in Spark in Scala
//sc -- SparkContext
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
var hadoopFileDataFrame =hiveContext.read.format("com.databricks.spark.csv").load(filePath)
2) Using Dataframe schema , create a table in Hive in Parquet format and load the data from dataframe to Hive Table.
Issue 1 : Dependency added in pom.xml for parquet-hive-bundle-1.6.0.jar .
Using following code:
var query = "CREATE TABLE Test(EMP_ID string,Organisation string,Org_Skill string,EMP_Name string)ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'parquet.hive.DeprecatedParquetInputFormat' OUTPUTFORMAT 'parquet.hive.DeprecatedParquetOutputFormat' TBLPROPERTIES ('PARQUET.COMPRESS'='SNAPPY')"
val dataFrame = hiveContext.sql(query)
The code hangs although the table is created but unable to fire select query on the table.
Issue 2:
hadoopFileDataFrame.registerTempTable("temp")
var query="CREATE TABLE TEST AS SELECT * FROM TEMP"
hiveContext.sql(query)
val dataFrame = hiveContext.sql("select * from test")
dataFrame.show()
Note: It successfully loads the data from dataframe to Hive Table as printed in the console logs. But when I check the Hive table using same Select Statement , there is no data in the table . What is the cause behind this?
How can I copy the data from dataframe to a Hive table and store it as Parquet file and perform dynamic partitioning of the data ?(ensuring that the data is correctly copied in the Hive table )
,
