Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Monitor Historical files and transfer to destination based on date

Monitor Historical files and transfer to destination based on date

Contributor

I have data flow which is checking folder /users/data/${now():format("yyyyMMdd")} and it would pick all files under this folder and upload to S3 /users/aws/${now():format("yyyyMMdd")}. Suppose i get file today which is 3 days old in the folder /users/data/20200331 i need to scan that folder and pick that file as well and move to S3 user /users/aws/20200331. 

Note : File names doesnt have date, Files are just dropped inside date folders.

 

@stevenmatison 

 

 

 

3 REPLIES 3
Highlighted

Re: Monitor Historical files and transfer to destination based on date

@Gubbi I think you would want to just add another check for now minus 1,2,3 days?  This seems most easy.   However I would wonder if you would just want to get everything from /users/data? Then the initial check doesn't care about now or today,  it just tries to get anything new in that folder.

 

 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: Monitor Historical files and transfer to destination based on date

Contributor

@stevenmatison : We dont have control on /users/data folder. Vendor would publish files in folder datewise. So there is no easy way to scan historical files without using minus expression. Minus would restrict for certain days, What if i have file a week back and i need to pick that one too.. There is no definite date when the old files would be pushed to that folder. 

Highlighted

Re: Monitor Historical files and transfer to destination based on date

@Gubbi   You need to figure out how to use ListFile to get everything, then check the dates/filenames/folders after the files arrived.  Regardless of what folders the files are in.   This way if a folder from 3 days ago appears tomorrow, you will still download the files.

 

Can you scan /user/data/ and take all folders and files?

 

 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Don't have an account?
Coming from Hortonworks? Activate your account here