Created 01-09-2017 02:14 PM
Hi, I have a scenario where I get a data file & control file. The data file has the actual data & the control file has the details about the data file, say filename,size etc. Below are the file names,
file_name - ABC.txt ctrl_file_name - CTRL_ABC.txt
I have to read the files from local filesystem & write into hdfs.
And say, I have the below flow in NiFi, List File -> Update attribute(file_name - ABC.txt,ctrl_file_name - CTRL_ABC.txt) Is there a way to combine these 2 attributes & read it using Fetch file processor & write it at once into HDFS?
Thanks!
Created 01-09-2017 02:25 PM
How do you plan to combine the "content" of these two files together?
This is the first question that needs to be addressed. You can use the mergeContent processor to merge the "content" of multiple FlowFiles using binary concatenation. If this is acceptable, then all you need to do is consume both your datafile and control file, use updateAttribute to extract the a common name from both filenames into new attribute, and finally use that new attribute as the "correlation Attribute Name" in the MergeContent processor.
You would also want to set the min number of entries to 2 in the mergeContent processor.
Matt
Created 03-08-2018 01:10 AM
Thank you. I solved it by using notify and wait processors. But I think the LookupAttribute solution sounds very interesting way for enrichment...Thanks Matt, I 've learned a lot from your responses.
Created 11-16-2022 04:06 AM
@mburgess - how did we achieve with this approach. do you have a sample?
Created 11-16-2022 09:08 PM
@kcc, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
Regards,
Vidya Sargur,