Not supported , as ListFTP or ListSFTP works on last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which Listftp already picked, then the file won't be picked with listftp logi, max or min file age property does not aline with current listing strategy.  ... View more

In response to your queries :   1,do you konw how to add "Minimum File Age" property in listftp processor ?  Ans : Not supported , as ListFTP or SFTP  works on last modified timestamp to pick newly modified files since it ran before. So, if a file is added with older last modified timestamp than the one which Listftp already picked, then the file won't be picked with listftp logic.  2.  listsftp can connect ftp server ?  i try ,but failed  Ans : Use ListFTP is FTP server is not secure. 3. What is the difference between listftp and listsftp?   We have two type of FTP servers, FTP and SFTP , SFTP uses a secure channel to transfer files while FTP doesn't, thus NiFi has FTP and SFTP processors.    To address your use case "file may be update anytime,  so i need the "Minimum File Age" property like listsftp processor get"   Ans : You need to change/update the logic on how files written at FTP server , If the same files getting updated/appended multiple time assuming write is not completed yet so try to rename the file with specific name pattern after append is completed  and only list/fetch the files which matches with rename pattern using File Filter Regex settings.          ... View more

Hi @ckumar - thanks for the reply.   I think that screenshot lacked info may be due to my permission issue.   Same issue has been posted by team mate here - https://community.cloudera.com/t5/Support-Questions/NiFi-Node-showing-2-nodes-and-not-respecting-node-down-fault/td-p/334221   Please take a look at the screenshot attachment there which has more info.   And i can login from all 3 nodes ui, and in all nodes it shows up only 1 & 2. ... View more

Can you also tell me how to ask data from a fixed date (period), not everything that there are in the bucket, thanks) ... View more

CountText will count lines (\r\n).   QueryRecord will count # of records, even if it is two records on a line ... View more

Tailfile stores local/cluster state to check where it has to start tailing again  or till what time it has already tailed the log file  like List* processor, but state information can not be used by other processor in NiFI to make their own work like flow files attribute value. Best way I could think of you have you think about buffer time/ retention time for log files and with in that retention time tailfile should be able to taill log files and delete the log files after their retention time, something  like whenever   ExecuteStreamCommand  run for delete it will only delete the files which has last modified time older then X number of days, assuming file which are older then X days is successfully tailed by Tailfile.   Thank You.  ... View more

@pacman  In addition to what @ckumar already shared: NiFI purposely leaves components visible to all on the canvas. but unless authorized to view those components, they will display as "ghost" implementations.   "Ghosted" components will not show any component names or classes on them.  They will only show stats.  Unauthorized users will be unable to view or modify the configuration.  User will also be unable to list or view data in connections (only see numbers of FlowFiles queued on a connection). The reason NiFi shows these ghosted components is to prevent multiple users from building their dataflows on top of one another.  It is very common for users to have multiple teams building their own dataflows, but then also have monitoring teams that may be authorized as "operators" across all dataflows. Or they may have some users that are members of multiple teams.   That means these users who can see more, would be left with potentially components layered on top of one another making management very difficult. The stats are there so even if a user can not view or modify a component, they can see where FlowFile backlogs are happening.  Since NiFi operates within a single JVM and every dataflow, no matter which user/team built them, is executed as the NiFi service user, everything must share the same system resources (various repos, Heap memory, disk  I/O, cpu, etc).   These stats provide useful information that one team can use to communicate to another team should resource utilization become an issue. NiFi's authorization model allows users to make very granular access decisions for every component.  Authorizations are inherited from the parent process group unless more granular policies are setup on a child component (processor, controller service, input/output port, sub-process group, etc..).  Hope this helps, Matt ... View more

yes it's quite right! i add a volumes on docker compose that connect directory on VM to docker-compose. Thank you so much! ... View more