Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Solved Go to solution
Highlighted

Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Contributor

@MattWho : 

I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. .

All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix.

For example, if today is 20200326, 

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file.

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder

/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.

 

Approach i followed :

ListFile: Extract only the files whose name ends with "gz.20200326"

UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"

 

Update.jpg

 

RouteOnAttribute: Extract only the file which matches

${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"

 

But when executing, looks like it's not matching my requirement and all files are matching.

Could you please assist me on this.

2 ACCEPTED SOLUTIONS

Accepted Solutions

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Master Collaborator

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route?  

 

I would recommend to simply the match in RouteOnAttribute:

 

${filename:endsWith(${today})}

 

If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven @ DFHZ

View solution in original post

Highlighted

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Master Collaborator

@Gubbi The next solution here is to just add each route for today, yesterday, day before yesterday.   Then route all 3 to the next proc.    Anything not matching won't be routed. 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven @ DFHZ

View solution in original post

5 REPLIES 5

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Master Collaborator

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route?  

 

I would recommend to simply the match in RouteOnAttribute:

 

${filename:endsWith(${today})}

 

If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven @ DFHZ

View solution in original post

Highlighted

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Contributor

@stevenmatison : I am not using GetFile here. i followed that approach you suggested but issue still persists. Below is dataflow i have been using. i need to route all matching files into target folder based on date. So for today's date 20200327, all files ending with _202003027 must reside in target folder 20200327.

So in my fetch file i am passing this /users/FIDS_OZ_DEV/Datastore/Bloomberg/backoffice/${now():format("yyyyMMdd")}

MFT.jpg

Highlighted

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Contributor

Ignore my previous reply. Solution which you provided is working. Thanks a lot.

Highlighted

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Contributor

@stevenmatison  : On similar thread, can i do match for 2-3 days and push files to respective folder.

Like ${filename:endsWith(${today -1})} and ${filename:endsWith(${today -2})}. Basically i want to keep checking 2 days previous files along with current date and push to respective folder based on date condition.

 

And how will i transfer those files previous day files to target date folder. Currently i am using /users/${now():format("yyyyMMdd")} to push today's file. If i have to do it for previous 2 days. how will it change.

Highlighted

Re: Transfer files based on the YYYYMMDD in their filename and create target folder with YYYYMMDD

Master Collaborator

@Gubbi The next solution here is to just add each route for today, yesterday, day before yesterday.   Then route all 3 to the next proc.    Anything not matching won't be routed. 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven @ DFHZ

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here