Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Best way to merge multi part file into single file?

Solved Go to solution

Best way to merge multi part file into single file?

New Contributor

We have huge data set in hdfs in multiple files and want to merge them all into single file to be used by our customers. We tried using hdfs getmerge command but running into OOM issues on edge node. Any other ways to achieve this merge functionality?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Best way to merge multi part file into single file?

Contributor

Use hadoop-streaming job (with single reducer) to merge all part files data to single hdfs file on cluster itself and then use hdfs get to fetch single file to local system.

$ hadoop jar /usr/hdp/2.3.2.0-2950/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.2.0-2950.jar \
                   -Dmapred.reduce.tasks=1 \
                   -input "/hdfs/input/dir" \
                   -output "/hdfs/output/dir" \
                   -mapper cat \
                   -reducer cat
5 REPLIES 5

Re: Best way to merge multi part file into single file?

Contributor

Use hadoop-streaming job (with single reducer) to merge all part files data to single hdfs file on cluster itself and then use hdfs get to fetch single file to local system.

$ hadoop jar /usr/hdp/2.3.2.0-2950/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.2.0-2950.jar \
                   -Dmapred.reduce.tasks=1 \
                   -input "/hdfs/input/dir" \
                   -output "/hdfs/output/dir" \
                   -mapper cat \
                   -reducer cat

Re: Best way to merge multi part file into single file?

New Contributor

Thanks! will try this.

Re: Best way to merge multi part file into single file?

New Contributor

If you are using spark then use below code:

sc.textFile("hdfs://...../part*).coalesce(1).saveAsTextFile("hdfs://...../filename)

This will merge all part files into one and save it again into hdfs location

Re: Best way to merge multi part file into single file?

New Contributor

Is there also an approach to combine snappy compressed files without decompressing/recompressing them? I have about 50 small files per hour, snappy compressed (framed stream, 65k chunk size) that I would like to combine to a single file, without recompressing (which should not be needed according to snappy documentation).

With above parameters the input files are decompressed (on-the-fly). I could of course recompress them during reduce, but that would be a waste of (CPU) resources.

Re: Best way to merge multi part file into single file?

New Contributor