Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

[#1]

@MattWho : 

I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. .

All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix.

For example, if today is 20200326, 

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file.

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.

Approach i followed :

ListFile: Extract only the files whose name ends with "gz.20200326"

UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"

RouteOnAttribute: Extract only the file which matches

${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"

But when executing, looks like it's not matching my requirement and all files are matching.

Could you please assist me on this.

[#2 · Accepted solution]

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route?  

I would recommend to simply the match in RouteOnAttribute:

${filename:endsWith(${today})}

If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

[#3 · Accepted solution]

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route?  

I would recommend to simply the match in RouteOnAttribute:

${filename:endsWith(${today})}

If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

[#4]

@stevenmatison : I am not using GetFile here. i followed that approach you suggested but issue still persists. Below is dataflow i have been using. i need to route all matching files into target folder based on date. So for today's date 20200327, all files ending with _202003027 must reside in target folder 20200327.

So in my fetch file i am passing this /users/FIDS_OZ_DEV/Datastore/Bloomberg/backoffice/${now():format("yyyyMMdd")}

[#5]

Ignore my previous reply. Solution which you provided is working. Thanks a lot.

[#6]

@stevenmatison  : On similar thread, can i do match for 2-3 days and push files to respective folder.

Like ${filename:endsWith(${today -1})} and ${filename:endsWith(${today -2})}. Basically i want to keep checking 2 days previous files along with current date and push to respective folder based on date condition.

And how will i transfer those files previous day files to target date folder. Currently i am using /users/${now():format("yyyyMMdd")} to push today's file. If i have to do it for previous 2 days. how will it change.

[#7 · Accepted solution]

@Gubbi The next solution here is to just add each route for today, yesterday, day before yesterday.   Then route all 3 to the next proc.    Anything not matching won't be routed.