question Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD in Support Questions

Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Gubbi — Fri, 27 Mar 2020 10:35:34 GMT

I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. .

All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix.

For example, if today is 20200326,

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file.

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.

Approach i followed :

ListFile: Extract only the files whose name ends with "gz.20200326"

UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"

RouteOnAttribute: Extract only the file which matches

${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"

But when executing, looks like it's not matching my requirement and all files are matching.

Could you please assist me on this.

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

stevenmatison — Fri, 27 Mar 2020 13:16:53 GMT

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route?

I would recommend to simply the match in RouteOnAttribute:

${filename:endsWith(${today})}

If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Gubbi — Fri, 27 Mar 2020 17:33:29 GMT

@stevenmatison : I am not using GetFile here. i followed that approach you suggested but issue still persists. Below is dataflow i have been using. i need to route all matching files into target folder based on date. So for today's date 20200327, all files ending with _202003027 must reside in target folder 20200327.

So in my fetch file i am passing this /users/FIDS_OZ_DEV/Datastore/Bloomberg/backoffice/${now():format("yyyyMMdd")}

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Gubbi — Fri, 27 Mar 2020 17:46:52 GMT

Ignore my previous reply. Solution which you provided is working. Thanks a lot.

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Gubbi — Wed, 01 Apr 2020 23:49:02 GMT

@stevenmatison : On similar thread, can i do match for 2-3 days and push files to respective folder.

Like ${filename:endsWith(${today -1})} and ${filename:endsWith(${today -2})}. Basically i want to keep checking 2 days previous files along with current date and push to respective folder based on date condition.

And how will i transfer those files previous day files to target date folder. Currently i am using /users/${now():format("yyyyMMdd")} to push today's file. If i have to do it for previous 2 days. how will it change.

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

stevenmatison — Thu, 02 Apr 2020 13:32:37 GMT

@Gubbi The next solution here is to just add each route for today, yesterday, day before yesterday. Then route all 3 to the next proc. Anything not matching won't be routed.