Member since
04-29-2016
192
Posts
20
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1636 | 07-14-2017 05:01 PM | |
2775 | 06-28-2017 05:20 PM |
01-19-2017
04:01 AM
2 Kudos
Hi All, Is there a reusable process group in NiFi ? When I say reusable, I mean similar to how a function works, you define it once and call/use it many times, but it is defined in only 1 place; and if you change the function's code, the change is reflected in all places where the function is used; When I try to reuse the same process group in multiple dataflows, I'm actually getting separate copies/instances of the process group; if I make a change in the reusable process group, I want the change to be reflected in all dataflows where this reusable process group is used; however, what I'm finding in NiFi is, you've to make the change in each instance of the process group. How do I get function like behavior when I want to reuse a process group ?
... View more
Labels:
- Labels:
-
Apache NiFi
01-15-2017
02:49 AM
1 Kudo
@Timothy Spann you were right on the money, increasing the JVM memory did the trick for me. Thanks.
... View more
01-13-2017
03:16 AM
Thanks @Matt for confirming. Sorry for not clarifying it better, my use case is to merge flow files that are from the same day, but I've both max and min # of entries set to 100, since I want to merge every 100 incoming flow files into a new merged file, as these files are small, less than 5kb each. So, I'm not trying to merge all flowfiles for the day into just 1 file.
... View more
01-12-2017
09:33 PM
Hello, couple of quick questions on Processor group with multiple input ports. 1) within the processor group, how do you distinguish between flowfiles that are coming from the various input ports 2) in data provenance page, is there a way to tell which flowfiles are from which input ports Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
01-12-2017
08:30 PM
thanks for clarifying @Bryan Bende, it makes sense.
... View more
01-12-2017
08:02 PM
Hi, Just wanted to know how NiFi's MergeContent processor would behave when all 3 of these properties are set - 1) Correlation Attribute Name, 2) Minimum Number of Entries and 3) Max Bin Age; in my use case, the Correlation Attribute Name is the flowfile date (without the time stamp), since I want to merge files that are from the same day; I set the minimum number of entries, because the flowfiles come at varying intervals throughout the day and I wanted to have similar number of flowfiles in each merged file; at midnight, when the day crosses over to next day, the bin may not get filled because of these 2 criteria (same day files and min # of entries); does Max Bin Age act as an override, when the other 2 conditions are not satisfied, and tell MergeContent to create a merged file with whatever files are in the bin ? I'm thinking yes, but I wanted to confirm. Thank you.
... View more
Labels:
- Labels:
-
Apache NiFi
01-12-2017
05:18 PM
@Matt, couple of follow up questions on Processor group with multiple input ports; 1) within the processor group, how do you distinguish between flowfiles that are coming from the various input ports. 2) in data provenance screen, is there a way to tell which flowfiles are from which input ports
... View more
01-12-2017
04:56 PM
1 Kudo
Thanks for the suggestion @Bryan Bende; I'll try that approach, in addition to the JVM and Cores that Timothy mentioned. I was thinking, if we batch files at ListenTCP, can we not Split them back to individual files with SplitText (the messages are text files) before passing them to our custom processor, minimizing the rework in the custom processor. Do you see any performance hit with this idea? Thanks.
... View more
01-12-2017
03:00 PM
Thanks @Bryan Bende for confirming, good to know; since I didn't see it in the documentation, I was concerned.
... View more