question Re: Move entire folder based on date to S3 in Support Questions

Move entire folder based on date to S3

Gubbi — Sun, 15 Mar 2020 02:38:46 GMT

Used case : Need to transfer entire folder of files based on date to S3.

Ex : Source Path (Linux) : /users/abc/20200312/gtry/gyyy.csv

S3 : /Users/datastore/20200312/gyyy.csv

Since dates keep changing everyday, and i need to build dataflow which would pick files from date folder

Re: Move entire folder based on date to S3

Gubbi — Mon, 16 Mar 2020 19:27:14 GMT

@Shu_ashu : I saw you had suggested similar pattern solution before. Could you please look into this and suggest approach.

https://community.cloudera.com/t5/Support-Questions/NiFi-Creating-the-output-directory-from-the-content-of-the/td-p/231335

Re: Move entire folder based on date to S3

Gubbi — Mon, 16 Mar 2020 19:31:12 GMT

@MattWho : Appreciate your inputs as well

Re: Move entire folder based on date to S3

MattWho — Tue, 17 Mar 2020 00:09:49 GMT

@Gubbi

Depending on which processor is being used to create your FlowFile from you source linux directory, you will likely have an "absolute.path" FlowFile attribute created on the FlowFile.

absolute.path = /users/abc/20200312/gtry/

You can pass that FlowFile to an UpdateAttribute processor which can use NiFi Expression Language (EL) to extract the date from that absolute path in to a new FlowFile attribute

Add new property (property name becomes new FlowFile attribute):

Property: Value:

pathDate ${absolute.path:getDelimitedField('4','/')}

The resulting FlowFile will have a new attribute:
pathDate = 20200312

Now you can use that FlowFile attribute later when writing to your target directory in S3.

I assume you would use the putS3Object processor for this?
If so, you can configure the "Object Key" property with the following:

/Users/datastore/${pathDate}/${filename}

NiFi EL will replace ${pathDate} with "20200312" and ${filename} will be replaced with "gyyy.csv".

Hope this helps you,

Matt