question Re: hive context save file in Archives of Support Questions (Read Only)

hive context save file

nanyim_alain — Tue, 07 Jun 2016 14:05:28 GMT

Hello

I work with Hive- Context to load and manipulate data in my orc format. I would now please know how to save in the hdfs file the results of a sql queries ?

Help me please?

Here is my Hive-Context code, I would like to save the contents of hive_context in a file on my hdfs :

Thanks you in advance

from pyspark.sql import HiveContext from pyspark import SparkContext
sc =SparkContext()
hive_context = HiveContext(sc) qvol = hive_context.table("<bdd_name>.<table_name>") qvol.registerTempTable("qvol_temp") hive_context.sql("select * from qvol_temp limit 10").show()

Re: hive context save file

rajkumar_singh — Tue, 07 Jun 2016 14:11:46 GMT

@alain TSAFACK please use saveAsHadoopFile while will write to hdfs

saveAsHadoopFile(<file-name>, <file output format>", compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec")

hive_context.write.format("orc").save("test_orc")

Re: hive context save file

nanyim_alain — Mon, 19 Aug 2019 10:14:35 GMT

Thank you. But here are the errors generated by the two attributes saveAsHadoopFile and .write.format:

This means that these two attributes are not recognized by HiveContext.

Thank you !!!

Re: hive context save file

rajkumar_singh — Tue, 07 Jun 2016 14:38:59 GMT

saveAsHadoopFile is applicable for RDD and is not for DF, can you try hive_context.write.format("orc").save("test_orc")

Re: hive context save file

nanyim_alain — Tue, 07 Jun 2016 14:49:11 GMT

I tried with hive_context.write.format("orc").save("test_orc") but I receive this error:

>>> hive_context.write.format("orc").save("hdfs://dev/datalake/app/des/dev/transformer/test_orc") Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'HiveContext' object has no attribute 'write'

Thanks

Re: hive context save file

rajkumar_singh — Tue, 07 Jun 2016 18:37:46 GMT

could you please modify your program in this way and see if you still see any excepton

from pyspark import SparkConf, SparkContext
from pyspark.sql import HiveContext
sc = SparkContext()
sqlContext = HiveContext(sc)
sqlContext.sql("select * from default.aaa limit 3").write.format("orc").save("test_orc2")

Re: hive context save file

nanyim_alain — Wed, 08 Jun 2016 15:42:50 GMT

Hello, thank you.

Here it goes. And i also found parquet file. Currently I am also looking to save as csv file and text if possible.

Cordially

Re: hive context save file

nanyim_alain — Thu, 09 Jun 2016 22:25:31 GMT

It's good I found !!!

thank you