Member since
09-20-2018
17
Posts
0
Kudos Received
0
Solutions
02-13-2023
06:45 AM
@jricogar Why not use the listHDFS processor? It retains state so that same HDFS files do not get listed multiple times. Just trying to understand your use case for using FetchHDFS without ListHDFS processor. Thanks, Matt
... View more
02-13-2022
11:14 PM
1 Kudo
@yamaga, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
... View more
10-18-2018
03:20 PM
@Pepelu Rico In the upcoming Apache NIFi 1.8 release, you may find following new capability will solve your use case issue here: https://jira.apache.org/jira/browse/NIFI-5406 Thanks, Matt
... View more
10-02-2018
01:10 PM
@Pepelu Rico Use MergeContent and RouteOnAttribute processors before MonitorActivity Processor. Configure MergeContent processor Minimum Number of Entries
5 So now processor will wait for 5 flowfiles and only if there are 5 flowfiles then processor merges all the flowfiles into 1 and transfer the flowfile to merged relationship. Use Max Bin Age property as a wildcard to force the bin to be merged, if we won't configure this property then processor will wait infinite time until it reaches 5 flowfiles. Then use the Merged Relationship from MergeContent processor to RouteOnAttribute processor RouteOnAttribute Configurations: MergeContent processor adds merge.count attribute to the flowfile so use that attribute to check if the value is 5 then give this relationship to MonitorActivity(run this processor at 12:00) processor. Add new property in RouteOnAttribute processor ${merge.count:equals(5)} Flow: 1.other processors 2.MergeContent -->use merged relation 3.RouteOnAttribute -->use new property relation 4.MonitorActivity
5.PutEmail in case if you want sequential merging then refer to this link for flow and configurations.
... View more
09-25-2018
10:00 AM
1 Kudo
@Pepelu Rico Yes we can validate the name of the file and size of file before ingesting into HDFS by using RouteOnAttribute processor. in NiFi we are having attributes to the flowfile as 1.${filename} //gets the filename of the flowfile
2.${fileSize} //gets the size in bytes of the flowfile In RouteOnAttribute processor add new property as valid_files ${filename:equlas("<required_file_name>"):and(${fileSize:gt(0)})} In the above expression we are using NiFi expression language and checking the filename value equlas(expression language function) to <required_file_name> and checking fileSize value is greater than 0, Then only the flowfiles will be tranferred into valid_files relation. Feed only the valid_files relation to PutHDFS processor, By using this relation we are storing only the files that satisfies the condition will be stored into HDFS. - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.
... View more
09-24-2018
02:24 PM
You should be able to use ValidateRecord for this, although your reader schema might need to treat the incoming field as an int/long so it will read in negative numbers successfully. Then your write schema can have that field with its correct type (date, e.g.) You can also use QueryRecord for this, it allows you to query the flow file fields using SQL, such as "SELECT * FROM FLOWFILE WHERE columnDate >= 0" or whatever is appropriate.
... View more