Reply
New Contributor
Posts: 2
Registered: ‎10-08-2016

Cannot make Hive to Write in Lzo format (compressed)

Hi All,

 

I am trying to insert data into a partitioned Hive Table from another unpartioned one. My intent is to get the inserted data from text to Lzo format. Below is the query and parameters that I am setting. Still I am unable to get that. Instead hive is writing the data in .deflate (which is the default compression mode). I have checked the environment and all the classes and referenced jar libraries are present. I can see the the correct classes being picked up in the map reduce steps (i.e. on the UI).

 

Is there something I am missing here? 

 

Thanks,

Apratim

 

set hive.exec.dynamic.partition=true;

set hive.exec.dynamic.partition.mode=nonstrict;

set hive.exec.max.created.files=100000;

set hive.exec.max.dynamic.partitions=150000;

set hive.exec.max.dynamic.partitions.pernode=10000;

set mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzoCodec;

set hive.exec.compress.output=true;

set mapreduce.output.fileoutputformat.compress=true;

SET mapreduce.output.fileoutputformat.compress.type=BLOCK;

 

INSERT INTO TEST_TABLE

PARTITION (table_id, p_event_type)

SELECT

  mm_imp_timestamp,

  mm_event_timestamp,

  mm_uuid,

  mm_adv_id,

  mm_s1,

  mm_s2,

....

...

mm_batch_id as table_id,

CASE

WHEN mm_pagetype='click' THEN 'click'

WHEN mm_pagetype='event' THEN 'event'

END p_event_type

FROM mini_impression_dump

DISTRIBUTE BY table_id, p_event_type;

Highlighted
New Contributor
Posts: 2
Registered: ‎10-08-2016

Re: Cannot make Hive to Write in Lzo format (compressed)

I resolved this issue by setting the below parameter in my query. Please ignore the question.

 

SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;

 

 

Thanks,

Apratim