Member since
02-25-2020
35
Posts
1
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1288 | 04-05-2020 01:51 PM | |
1573 | 03-15-2020 02:04 PM | |
4575 | 03-04-2020 12:15 PM |
03-27-2020
10:33 AM
@stevenmatison : I am not using GetFile here. i followed that approach you suggested but issue still persists. Below is dataflow i have been using. i need to route all matching files into target folder based on date. So for today's date 20200327, all files ending with _202003027 must reside in target folder 20200327. So in my fetch file i am passing this /users/FIDS_OZ_DEV/Datastore/Bloomberg/backoffice/${now():format("yyyyMMdd")}
... View more
03-26-2020
10:40 PM
@MattWho :
I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. .
All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix.
For example, if today is 20200326,
/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file.
/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder
/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.
Approach i followed :
ListFile: Extract only the files whose name ends with "gz.20200326"
UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"
RouteOnAttribute: Extract only the file which matches
${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"
But when executing, looks like it's not matching my requirement and all files are matching.
Could you please assist me on this.
... View more
Labels:
- Labels:
-
Apache NiFi
03-26-2020
10:22 PM
@shuhei_shogen : I had similar used case but when i tried to reproduce the same approach its not working.My file name is like equity_asia2.dif.gz.20200324 and i want these files to be in target folder 20200324. In Update Attribute i have used ${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})}. But it doesnt seem to be working. Could you please check and assist me on this.
... View more
03-16-2020
12:27 PM
@Shu_ashu : I saw you had suggested similar pattern solution before. Could you please look into this and suggest approach. https://community.cloudera.com/t5/Support-Questions/NiFi-Creating-the-output-directory-from-the-content-of-the/td-p/231335
... View more
03-15-2020
02:04 PM
Issue has been resolved.
... View more
03-15-2020
11:22 AM
I have NAS mounted on Linux server and have to copy files from location to S3, i am using ListFile→FetchFile-→PutS3.
While reading files from NAS mount location, i am getting error saying file not found but i see all required files are present at location. Is there any special configuration which needs to be set while reading files from NAS drive.
... View more
Labels:
- Labels:
-
Apache NiFi
03-13-2020
01:42 PM
Used case : Need to transfer entire folder of files based on date to S3.
Ex : Source Path (Linux) : /users/abc/20200312/gtry/gyyy.csv
S3 : /Users/datastore/20200312/gyyy.csv
Since dates keep changing everyday, and i need to build dataflow which would pick files from date folder
... View more
Labels:
- Labels:
-
Apache NiFi
03-04-2020
12:15 PM
1 Kudo
I got solution for this. Had to use expression language in Object Key to fetch date from file and it worked. Below is expression :${filename:substringAfter('.gz.')}/${filename}
... View more
03-03-2020
10:42 PM
@AustinLiu Just to clarify, i am transferring files from linux box to S3.
... View more
- « Previous
-
- 1
- 2
- Next »