question Re: How to validate which compression technique has been applied to hive? in Archives of Support Questions (Read Only)

How to validate which compression technique has been applied to hive?

payal_patel — Thu, 21 Jul 2016 16:59:32 GMT

I have create table in hive and specified compression as gzip during table creation

I am not able to find way that gzip has been applied on newly created table

is there any command which specify compression technique applied on hive.

Re: How to validate which compression technique has been applied to hive?

ssubhas — Thu, 21 Jul 2016 17:01:44 GMT

@payal patel

The compression of the table can be validated by checking the definition of the hive table using the command:

hive> describe formatted <hive_table>;

Re: How to validate which compression technique has been applied to hive?

rpathak — Thu, 21 Jul 2016 17:04:32 GMT

@payal patel

What you see when you run show create table <tablename>; ?

Re: How to validate which compression technique has been applied to hive?

payal_patel — Thu, 21 Jul 2016 17:20:36 GMT

I am running hive query using HIve view of Ambari and executing below query

CREATE TABLE t11 STORED AS parquet TBLPROPERTIES("parquet.COMPRESS"="gzip") AS SELECT * FROM abc;

Here abc table already contain some data

Post execution, it shows that query status as "Succeed"

then I verified data by executing "select * from t11".I am able to see data.

I want to verify that gzip has been applied or not.so I did not find way yet

Re: How to validate which compression technique has been applied to hive?

payal_patel — Thu, 21 Jul 2016 17:23:23 GMT

I am able to see that compressed as No. I think I need to set configuration for compression separately in hive

Re: How to validate which compression technique has been applied to hive?

payal_patel — Thu, 21 Jul 2016 17:23:54 GMT

there is message only that query runs "succeed"

Re: How to validate which compression technique has been applied to hive?

rpathak — Thu, 21 Jul 2016 17:25:55 GMT

show create table t11;

What you get in output?

Re: How to validate which compression technique has been applied to hive?

payal_patel — Thu, 21 Jul 2016 17:36:21 GMT

able to run ..Thanks

Re: How to validate which compression technique has been applied to hive?

payal_patel — Thu, 21 Jul 2016 17:37:45 GMT

I am able to resolve it. issue with create and select together.

I've created table first then inserted data, now able to see gzip as tableproperties in desc command

Re: How to validate which compression technique has been applied to hive?

payal_patel — Fri, 22 Jul 2016 01:07:28 GMT

getting below output

CREATE TABLE `t11`( `organisation` string, `org_skill` string, `emp_name` string, `pageurl` string, `cookie` string, `referurl` string, `time` string, `employeeid` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/t11' TBLPROPERTIES ( 'COLUMN_STATS_ACCURATE'='true', 'numFiles'='1', 'numRows'='55', 'parquet.COMPRESS'='gzip', 'rawDataSize'='440', 'totalSize'='2046', 'transient_lastDdlTime'='1469095673')

Re: How to validate which compression technique has been applied to hive?

payal_patel — Fri, 22 Jul 2016 01:17:05 GMT

Thanks for reply..

I've one more question

the table properties showing that gzip has been applied,so i've created table post disabled hive compression.(disabled partition by setting as parquet.compress="none" in create table syntax and set hive.exec.compress.output"="false" )

but when i execute below command on both tables then i dont see any difference for size of withcompression or withoutcompression hive table.

command: $> hadoop fs -du -h hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/tt1.

I've set below property in hive while enabling compression $>hive.exec.compress.output","true"