Reply
Highlighted
Anonymous
Posts: 0

Saving snappy compressed Avro data without compression?

[ Edited ]

I have a Snappy compressed avro data file that I imported through sqoop

 

1.png

 

save it uncompressed (Spark 1.6)

 

>>> sqlContext.getConf("spark.sql.avro.compression.codec","")
u''

>>> df = sqlContext.read.format("com.databricks.spark.avro").load("/user/cloudera/myavro/cat_avro3/")

>>> sqlContext.setConf("spark.sql.avro.compression.codec","uncompressed")
18/01/23 04:58:49 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.1.0-cdh5.12.0
18/01/23 04:58:49 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException

>>> df.write.format("com.databricks.spark.avro").save("/user/cloudera/myavro/uc1/")

 avro-tools shows that the codec is null but when I open the file in filebrowser, it shows Output rendered from compressed avro file

22.png

 

3.png

 

 

Which one is reliable?

 

thanks

Announcements