Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NIFI : how can i merge two csv from different source into one csv file .(merge Record)

NIFI : how can i merge two csv from different source into one csv file .(merge Record)

Explorer

Hi All,

 

I have an urgent requirement. 

Suppose i have two csvs:

one csv - a,b,c,d

second csv - e,f,g,h

 

I want to merge both csv so as to get one csv with output as a,b,c,d,e,f,g,h.

 

Both the csvs are getting generated from sql via executesqlrecord processor.

Request to help as soon as possible.

2 REPLIES 2
Highlighted

Re: NIFI : how can i merge two csv from different source into one csv file .(merge Record)

Expert Contributor

I'm not aware of native way of doing this in NiFi, may be someone else will be able to chime in. Couple pieces of information that would help here is:

  1. How big are the CSV files?
  2. Do the CSV files have any common column to join on or is the order of records always ensured?

 

Couple alternatives for you:

  • Are the two CSVs being generated from the same database? If so, why not do a single execute SQL processor that does the selection of all columns for you.
  • If there are common IDs and the CSV tables are on the small side, you can load one of them into Lookup Service and do a lookup with LookupRecord.
  • Final solution is to write a shell or Python script to perform the operation and execute it from NiFi with ExecuteProcess
Highlighted

Re: NIFI : how can i merge two csv from different source into one csv file .(merge Record)

Explorer

Hi @aakulov

 

I have around 4 csv files which are having 5-8 columns and yes they do have time column in common.

I am able to merge csv but in different rows not in same one.

 

Request others to help as well

 

Don't have an account?
Coming from Hortonworks? Activate your account here