Reply
Expert Contributor
Posts: 87
Registered: ‎09-17-2014

Double read for lzma compression codec

Hi experts!

 

i will very appreciate if somebody could help me to understand - why i read in two rimes more data with lzma codec (https://github.com/yongtang/hadoop-xz).

I've started to play with this and met interesting thing. When I try to proceed data with lzma i read in two times more data then i'm actually have on the HDFS.

For example, hadoop client (hadoop fs -du) shows some numbers like 100GB.

then i run MR (like select count(1) ) over this data and check MR counters and find "HDFS bytes read" two times more (like 200GB).

In case of gzip and bzip2 codecs hadoop client file size and MR counters are the similar

 

Highlighted
Cloudera Employee
Posts: 285
Registered: ‎01-16-2014

Re: Double read for lzma compression codec

The codec is responsible for the reads and you will need to talk to the creator of the codec to provide you with the information on why this is happening.

 

Wilfred

Announcements