Support Questions

rohit_ravishank · ‎01-09-2017

Hi, I have a scenario where I get a data file & control file. The data file has the actual data & the control file has the details about the data file, say filename,size etc. Below are the file names,

file_name - ABC.txt ctrl_file_name - CTRL_ABC.txt

I have to read the files from local filesystem & write into hdfs.

And say, I have the below flow in NiFi, List File -> Update attribute(file_name - ABC.txt,ctrl_file_name - CTRL_ABC.txt) Is there a way to combine these 2 attributes & read it using Fetch file processor & write it at once into HDFS?

Thanks!

MattWho · ‎01-09-2017

@Rohit Ravishankar

How do you plan to combine the "content" of these two files together?

This is the first question that needs to be addressed. You can use the mergeContent processor to merge the "content" of multiple FlowFiles using binary concatenation. If this is acceptable, then all you need to do is consume both your datafile and control file, use updateAttribute to extract the a common name from both filenames into new attribute, and finally use that new attribute as the "correlation Attribute Name" in the MergeContent processor.

You would also want to set the min number of entries to 2 in the mergeContent processor.

Matt

View solution in original post

skhaksho · ‎03-08-2018

Thank you. I solved it by using notify and wait processors. But I think the LookupAttribute solution sounds very interesting way for enrichment...Thanks Matt, I 've learned a lot from your responses.

kcc · ‎11-16-2022

@mburgess - how did we achieve with this approach. do you have a sample?

VidyaSargur · ‎11-16-2022

@kcc, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

Regards,

Vidya Sargur,
Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Community Guidelines
How to use the forum

Support Questions

Combining Attributes in NiFi