question Can you create a hive table in ORC Format from SparkSQL directly in Archives of Support Questions (Read Only)

Can you create a hive table in ORC Format from SparkSQL directly

TimothySpann — Wed, 01 Jun 2016 02:50:41 GMT

I've done so with a sqlContext.sql ("create table...") then a sqlContext.sql("insert into")

but a dataframe.write.orc will produce an ORC file that cannot be seen as hive.

What are all the ways to work with ORC from Spark?

Re: Can you create a hive table in ORC Format from SparkSQL directly

jyadav — Wed, 01 Jun 2016 02:56:16 GMT

@Timothy Spann

Did you tried with this syntax?

var Rddtb= objHiveContext.sql("select * from sample")
val dfTable = Rddtb.toDF()
dfTable.write.format("orc").mode(SaveMode.Overwrite).saveAsTable("db1.test1")

Re: Can you create a hive table in ORC Format from SparkSQL directly

TimothySpann — Wed, 01 Jun 2016 03:21:02 GMT

@Jitendra Yadav that worked for me in zeppelin and the data looks good.

Re: Can you create a hive table in ORC Format from SparkSQL directly

mkumar13 — Wed, 08 Jun 2016 15:49:50 GMT

Answer to your Question "What are all the ways to work with ORC from Spark?"

I am using spark-sql and have created ORC table as well as other formats and found no issue.

Re: Can you create a hive table in ORC Format from SparkSQL directly

uday_vakalapudi — Mon, 19 Aug 2019 11:11:56 GMT

The Optimized Row Columnar (ORC) file format provides a highly efficient way to store Hive data.

It just like a File to store group of rows called stripes, along with auxiliary information in a file footer. It just a storage format, nothing to do with ORC/Spark.