If I use a pig script like the one described below, I am able to leverage mapreduce to compress a ton of data and I get a pretty good ratio. However, when I try to de-compress the data, I lose the individual files. For example, if my original, uncompressed folder has a.dat through z.dat, the compressed folder will have something like part-m-00001.bz2, part-m-00002.bz2, etc That's fine. But then, when I try to do the same thing in reverse, to get back my original content, I just get larger files that look like part-m-00001, part-m-00002, etc. Is there a way to leverage our cluster to compress HDFS files in such a way that I can get back the original files - including the file name? Thanks! set
output.compression.enabled true; set
output.compression.codec org.apache.hadoop.io.compress.BZip2Codec; InputFiles = LOAD
'/my/hdfs/path/' using PigStorage(); STORE InputFiles INTO
'/my/hdfs/path_compressed/' USING PigStorage();
... View more
