Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

merge csv files based on a column timestamp to get one file

merge csv files based on a column timestamp to get one file

Explorer

i have 4 csv files , i want to join and merge these files into one files based on a column timestamps to get one file.using spark or hadoop Please any help would be appreciated

2 REPLIES 2
Highlighted

Re: merge csv files based on a column timestamp to get one file

Sol 1: Reduce side join

create separate mappers for all 4 csv files and produce the time stamp as key from all mappers and remaining fields + tag field to represent from which file it is returned as value.

handle them in reduce side..

Sol 2: Map side join (if 3 files are small)

add 3 csv files into distributed cache and merge them with large csv file in mapper.

Highlighted

Re: merge csv files based on a column timestamp to get one file

Expert Contributor

hi @Maher Hattabi

I am seeing a similar question of yours in the link below.

Here is one where i answered the question combining any files whether it be csv or txt

https://community.hortonworks.com/questions/85230/erge-csv-files-in-one-file.html#answer-85245

Don't have an account?
Coming from Hortonworks? Activate your account here