Member since
02-25-2020
35
Posts
1
Kudos Received
3
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2011 | 04-05-2020 01:51 PM | |
| 2526 | 03-15-2020 02:04 PM | |
| 6217 | 03-04-2020 12:15 PM |
03-26-2020
10:40 PM
@MattWho :
I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. .
All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix.
For example, if today is 20200326,
/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file.
/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder
/tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.
Approach i followed :
ListFile: Extract only the files whose name ends with "gz.20200326"
UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"
RouteOnAttribute: Extract only the file which matches
${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"
But when executing, looks like it's not matching my requirement and all files are matching.
Could you please assist me on this.
... View more
Labels:
- Labels:
-
Apache NiFi
03-16-2020
12:27 PM
@Shu_ashu : I saw you had suggested similar pattern solution before. Could you please look into this and suggest approach. https://community.cloudera.com/t5/Support-Questions/NiFi-Creating-the-output-directory-from-the-content-of-the/td-p/231335
... View more
03-15-2020
02:04 PM
Issue has been resolved.
... View more
03-15-2020
11:22 AM
I have NAS mounted on Linux server and have to copy files from location to S3, i am using ListFile→FetchFile-→PutS3.
While reading files from NAS mount location, i am getting error saying file not found but i see all required files are present at location. Is there any special configuration which needs to be set while reading files from NAS drive.
... View more
Labels:
- Labels:
-
Apache NiFi
03-13-2020
01:42 PM
Used case : Need to transfer entire folder of files based on date to S3.
Ex : Source Path (Linux) : /users/abc/20200312/gtry/gyyy.csv
S3 : /Users/datastore/20200312/gyyy.csv
Since dates keep changing everyday, and i need to build dataflow which would pick files from date folder
... View more
Labels:
- Labels:
-
Apache NiFi
03-04-2020
12:15 PM
1 Kudo
I got solution for this. Had to use expression language in Object Key to fetch date from file and it worked. Below is expression :${filename:substringAfter('.gz.')}/${filename}
... View more
03-03-2020
10:42 PM
@AustinLiu Just to clarify, i am transferring files from linux box to S3.
... View more
03-03-2020
10:31 PM
@AustinLiu : But i need to transfer file abcd_20200303 to S3 folder 20200303 and respectively based on dates. Every day when the files arrive my processor should identify file based on date and push it to respective date folder in S3.
... View more
03-03-2020
08:00 PM
I have used case where i am reading files with timestamp and these files has to be transferred to S3 and create folder with respective dates. Ex: file names abcd.out.gz.20200303 , abcd.out.gz.20200302
and the file abcd.out.gz.20200303 need to be in S3 under /data/20200303
and file abcd.out.gz.20200302 under /data/20200302.
How can i achieve this in NiFi.
... View more
Labels:
- Labels:
-
Apache NiFi
- « Previous
-
- 1
- 2
- Next »