Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot make Hive to Write in Lzo format (compressed)

Cannot make Hive to Write in Lzo format (compressed)

New Contributor

Hi All,

 

I am trying to insert data into a partitioned Hive Table from another unpartioned one. My intent is to get the inserted data from text to Lzo format. Below is the query and parameters that I am setting. Still I am unable to get that. Instead hive is writing the data in .deflate (which is the default compression mode). I have checked the environment and all the classes and referenced jar libraries are present. I can see the the correct classes being picked up in the map reduce steps (i.e. on the UI).

 

Is there something I am missing here? 

 

Thanks,

Apratim

 

set hive.exec.dynamic.partition=true;

set hive.exec.dynamic.partition.mode=nonstrict;

set hive.exec.max.created.files=100000;

set hive.exec.max.dynamic.partitions=150000;

set hive.exec.max.dynamic.partitions.pernode=10000;

set mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzoCodec;

set hive.exec.compress.output=true;

set mapreduce.output.fileoutputformat.compress=true;

SET mapreduce.output.fileoutputformat.compress.type=BLOCK;

 

INSERT INTO TEST_TABLE

PARTITION (table_id, p_event_type)

SELECT

  mm_imp_timestamp,

  mm_event_timestamp,

  mm_uuid,

  mm_adv_id,

  mm_s1,

  mm_s2,

....

...

mm_batch_id as table_id,

CASE

WHEN mm_pagetype='click' THEN 'click'

WHEN mm_pagetype='event' THEN 'event'

END p_event_type

FROM mini_impression_dump

DISTRIBUTE BY table_id, p_event_type;

1 REPLY 1

Re: Cannot make Hive to Write in Lzo format (compressed)

New Contributor

I resolved this issue by setting the below parameter in my query. Please ignore the question.

 

SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;

 

 

Thanks,

Apratim