Member since
10-17-2017
2
Posts
0
Kudos Received
0
Solutions
11-20-2017
10:12 AM
Thanks for your quick reply and sorry for late. I tried your suggestion and was successfully able to do what I want to do. What I did: ListFile: Extract only the files whose name ends with ".tsv" UpdateAttribute: Substitute YYYYMMDD into an Attribute "today" RouteOnAttribute: Extract only the file which matches `${filename:matches(${today:prepend('[^\.].*_'):append('\d{6}.tsv$')})}` (where $filename is the target file name) and pass it as Relationship "target_file"
... View more
11-14-2017
02:42 PM
What I want to do: I'm finding a way to get the content of a file based on its filename. All of the target files are in the same directory, but I'd like to select only the files which has "_{Today's YYYYMMDD}.tsv" as their postfix. For example, if today is 20171113, /same/dir/testfile_20171113.tsv -> OK. I'd like to ingest this file. /same/dir/testfile2_20171113.tsv -> OK. This one is also a target. /same/dir/testfile_20171114.tsv -> NG because this YYYYMMDD is not today. /same/dir/testfile_2017111.tsv -> NG because the timestamp is not in the format of YYYYMMDD. /same/dir/testfile_20171113.tsv.processed -> NG because the filename does not end with ".tsv". /another/dir/testfile_20171113.tsv -> NG because this file is on another directory What I have investigated: I have gone through these docs and tried ListFile Processor and GetFile Processor, https://community.hortonworks.com/questions/38120/how-to-get-files-based-on-the-time-stamp-in-nifi.html https://community.hortonworks.com/questions/39553/how-to-get-files-based-on-dates-in-nifi.html I tried to input "[^\.].*_${now():format('yyyyMMdd')}\.tsv$" as "File Filter", but got an error which said "Not a valid Java Regular Expression". As far as I checked, "File Filter" on both of ListFile and GetFile use StandardValidators.REGULAR_EXPRESSION_VALIDATOR as their validators, and unfortunately this validator does not interpret Nifi Expression Language. https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/processor/util/StandardValidators.java#L363 https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/processor/util/StandardValidators.java#L619 Question * Is there any way to inject an expression language into StandardValidators.REGULAR_EXPRESSION_VALIDATOR? * If not, is there any other way to solve this problem? Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi