i have 4 csv files , i want to join and merge these files into one files based on a column timestamps to get one file.using spark or hadoop Please any help would be appreciated
Sol 1: Reduce side join
create separate mappers for all 4 csv files and produce the time stamp as key from all mappers and remaining fields + tag field to represent from which file it is returned as value.
handle them in reduce side..
Sol 2: Map side join (if 3 files are small)
add 3 csv files into distributed cache and merge them with large csv file in mapper.
hi @Maher Hattabi
I am seeing a similar question of yours in the link below.
Here is one where i answered the question combining any files whether it be csv or txt