Member since
01-18-2022
14
Posts
0
Kudos Received
0
Solutions
02-13-2022
05:45 PM
The QueryRecord config like this: I add a new route relationship: select 907 fileds from the merge flowfile. the configuation of nifi like this in bootstrap.conf the select like this: select 907 fileds FROM ( SELECT * FROM FLOWFILE WHERE d = 'devicedata' ) devicedata LEFT JOIN ( SELECT * FROM FLOWFILE WHERE d = 'deviceinfo' ) deviceinfo ON devicedata.manufacturer_oui = deviceinfo.manufacturer_oui AND devicedata.serial_number = deviceinfo.serial_number
... View more
02-11-2022
03:33 AM
Hello Zhangliang, For the more efficient tuning of the content repository archive you can use these pages (you might have already known them): https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418 https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html#deeper-view-provenance-log-files also 1] - Add the following property from Ambari under nifi.properties section to a value of 75% nifi.content.repository.archive.backpressure.percentage to 75% 2] Increase the disk space.
... View more
01-26-2022
06:25 AM
Hello, Most likely because on your CSV Reader you have: Treat First Line as Header = false ( default ) Change that to true
... View more
01-21-2022
09:46 AM
1 Kudo
Without knowing how the FTP directory is populated, you might have an issue with the ListFTP state and might consider changing its its Listing Strategy, However assuming that is no issue and you want to only perform an action based on any file with a naming structure like this: xxxx-xxxx-xxxx-xxxx-20220121110000-xxxx-xxxx.csv and only if it is older than two hours from "now" I would run the output of your ListFTP to an UpdateAttribute and add these 2 properties: property name fileTime value ${filename:getDelimitedField(5,'-'):trim():toDate('yyyyMMddHHmmss'):toNumber()} property name timeNow value ${now():toNumber():minus(7200000)} Then route that to a RouteOnAttribute and add a new property: property name value 2 hours old ${fileTime:le(${timeNow})} Then you can drag that connection to follow on processing and the unmatched connection to other processing or terminate it. Explanation: filelName ${filename:getDelimitedField(5,'-'):trim():toDate('yyyyMMddHHmmss'):toNumber()} This grabs the time out of your filename and converts it to a Date object so it can be converted to its epoch representation timeNow ${now():toNumber():minus(7200000)} Sets a value 2 hours ago from current time ${fileTime:le(${timeNow})} If attribute fileName is less than or equal to timeNow it means that it is 2 hours old.
... View more
01-21-2022
03:15 AM
Not supported , as ListFTP or ListSFTP works on last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which Listftp already picked, then the file won't be picked with listftp logi, max or min file age property does not aline with current listing strategy.
... View more
01-21-2022
03:08 AM
In response to your queries : 1,do you konw how to add "Minimum File Age" property in listftp processor ? Ans : Not supported , as ListFTP or SFTP works on last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which Listftp already picked, then the file won't be picked with listftp logic. 2. listsftp can connect ftp server ? i try ,but failed Ans : Use ListFTP is FTP server is not secure. 3. What is the difference between listftp and listsftp? We have two type of FTP servers, FTP and SFTP , SFTP uses a secure channel to transfer files while FTP doesn't, thus NiFi has FTP and SFTP processors. To address your use case "file may be update anytime, so i need the "Minimum File Age" property like listsftp processor get" Ans : You need to change/update the logic on how files written at FTP server , If the same files getting updated/appended multiple time assuming write is not completed yet so try to rename the file with specific name pattern after append is completed and only list/fetch the files which matches with rename pattern using File Filter Regex settings.
... View more