- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How do I create an ORC Hive table from Spark?
- Labels:
-
Apache Hive
-
Apache Spark
Created ‎11-19-2015 03:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm currently using Spark 1.4 and I'm loading some data into a DataFrame using jdbc:
val jdbcDF = sqlContext.load("jdbc", options)
How can I save the jdbcDF DataFrame to a Hive table using the ORC file format?
Created ‎11-19-2015 03:07 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
df.write.format("orc") will get you there.
See: http://hortonworks.com/blog/bringing-orc-support-into-apache-spark/ or http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_spark-guide/content/ch_orc-spark.html
Created ‎03-25-2016 03:45 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Brandon Wilson I tried your suggestion it creates the hive table but I get this error:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table.
and it does not load data into my table. do you have any idea how to solve this?
Created ‎12-17-2015 02:21 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Divya,
What is the user account when DF is used to create the external Hive table?
What is the user account when you try to see the table in Hive (& did you use HiveCli or Hive/Beeline or some ODBC tool?)
Created ‎12-18-2015 06:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@vshukla I am logging in as hdfs user on HDP 2.3.2 sandbox
and using the same account to see tables in hive.Yes , I am using hive CLI and even browsed HDFS files through Ambari .Couldnt see any tables created.

- « Previous
-
- 1
- 2
- Next »