Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

avatar
Contributor

@MattWho : 

I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. .

All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix.

For example, if today is 20200326, 

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file.

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.

 

Approach i followed :

ListFile: Extract only the files whose name ends with "gz.20200326"

UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"

 

Update.jpg

 

RouteOnAttribute: Extract only the file which matches

${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"

 

But when executing, looks like it's not matching my requirement and all files are matching.

Could you please assist me on this.

2 ACCEPTED SOLUTIONS

avatar
Super Guru

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route?  

 

I would recommend to simply the match in RouteOnAttribute:

 

${filename:endsWith(${today})}

 

If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

View solution in original post

avatar
Super Guru

@Gubbi The next solution here is to just add each route for today, yesterday, day before yesterday.   Then route all 3 to the next proc.    Anything not matching won't be routed. 

View solution in original post

5 REPLIES 5

avatar
Super Guru

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route?  

 

I would recommend to simply the match in RouteOnAttribute:

 

${filename:endsWith(${today})}

 

If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

avatar
Contributor

@stevenmatison : I am not using GetFile here. i followed that approach you suggested but issue still persists. Below is dataflow i have been using. i need to route all matching files into target folder based on date. So for today's date 20200327, all files ending with _202003027 must reside in target folder 20200327.

So in my fetch file i am passing this /users/FIDS_OZ_DEV/Datastore/Bloomberg/backoffice/${now():format("yyyyMMdd")}

MFT.jpg

avatar
Contributor

Ignore my previous reply. Solution which you provided is working. Thanks a lot.

avatar
Contributor

@stevenmatison  : On similar thread, can i do match for 2-3 days and push files to respective folder.

Like ${filename:endsWith(${today -1})} and ${filename:endsWith(${today -2})}. Basically i want to keep checking 2 days previous files along with current date and push to respective folder based on date condition.

 

And how will i transfer those files previous day files to target date folder. Currently i am using /users/${now():format("yyyyMMdd")} to push today's file. If i have to do it for previous 2 days. how will it change.

avatar
Super Guru

@Gubbi The next solution here is to just add each route for today, yesterday, day before yesterday.   Then route all 3 to the next proc.    Anything not matching won't be routed.