Created 11-19-2015 03:03 AM
I'm currently using Spark 1.4 and I'm loading some data into a DataFrame using jdbc:
val jdbcDF = sqlContext.load("jdbc", options)
How can I save the jdbcDF DataFrame to a Hive table using the ORC file format?
Created 11-19-2015 03:07 AM
df.write.format("orc") will get you there.
See: http://hortonworks.com/blog/bringing-orc-support-into-apache-spark/ or http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_spark-guide/content/ch_orc-spark.html
Created 03-25-2016 03:45 PM
@Brandon Wilson I tried your suggestion it creates the hive table but I get this error:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table.
and it does not load data into my table. do you have any idea how to solve this?
Created 12-17-2015 02:21 AM
Divya,
What is the user account when DF is used to create the external Hive table?
What is the user account when you try to see the table in Hive (& did you use HiveCli or Hive/Beeline or some ODBC tool?)
Created 12-18-2015 06:26 AM
@vshukla I am logging in as hdfs user on HDP 2.3.2 sandbox
and using the same account to see tables in hive.Yes , I am using hive CLI and even browsed HDFS files through Ambari .Couldnt see any tables created.