Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Saving snappy compressed Avro data without compression?

Saving snappy compressed Avro data without compression?

I have a Snappy compressed avro data file that I imported through sqoop




save it uncompressed (Spark 1.6)


>>> sqlContext.getConf("spark.sql.avro.compression.codec","")

>>> df ="com.databricks.spark.avro").load("/user/cloudera/myavro/cat_avro3/")

>>> sqlContext.setConf("spark.sql.avro.compression.codec","uncompressed")
18/01/23 04:58:49 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.1.0-cdh5.12.0
18/01/23 04:58:49 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException

>>> df.write.format("com.databricks.spark.avro").save("/user/cloudera/myavro/uc1/")

 avro-tools shows that the codec is null but when I open the file in filebrowser, it shows Output rendered from compressed avro file






Which one is reliable?



Don't have an account?
Coming from Hortonworks? Activate your account here