We have huge data set in hdfs in multiple files and want to merge them all into single file to be used by our customers. We tried using hdfs getmerge command but running into OOM issues on edge node. Any other ways to achieve this merge functionality?
Is there also an approach to combine snappy compressed files without decompressing/recompressing them? I have about 50 small files per hour, snappy compressed (framed stream, 65k chunk size) that I would like to combine to a single file, without recompressing (which should not be needed according to snappy documentation).
With above parameters the input files are decompressed (on-the-fly). I could of course recompress them during reduce, but that would be a waste of (CPU) resources.