Support Questions

Find answers, ask questions, and share your expertise

merge csv files based on a column timestamp to get one file


i have 4 csv files , i want to join and merge these files into one files based on a column timestamps to get one file.using spark or hadoop Please any help would be appreciated


Sol 1: Reduce side join

create separate mappers for all 4 csv files and produce the time stamp as key from all mappers and remaining fields + tag field to represent from which file it is returned as value.

handle them in reduce side..

Sol 2: Map side join (if 3 files are small)

add 3 csv files into distributed cache and merge them with large csv file in mapper.

Expert Contributor

hi @Maher Hattabi

I am seeing a similar question of yours in the link below.

Here is one where i answered the question combining any files whether it be csv or txt