Member since
01-20-2017
18
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2521 | 03-06-2017 12:13 PM |
03-06-2017
12:52 PM
@adrian white Try to play around with attributes that are output by the merge content processor - merge.count and merge.bin.age. Depending on merge.bin.age value you can root the flow forward or to "/dev/null" equivalent.
... View more
03-06-2017
12:13 PM
1 Kudo
Have you tried using MergeContent processor? The whole configuration depends on how your flow file looks like, but this might be a good starting point.
... View more
02-24-2017
03:45 PM
@Karthik Narayanan File is local on NiFi box. I know that ExecuteProcess would be an option but I'd like to avoid saving the file on the disk. How about ExecuteStreamCommand and something like: sed '1d' simple.tsv > noHeader.tsv but in a way: sed '1d' myFlow > myFlow. How I can reference myFlow in this command?
... View more
02-24-2017
03:21 PM
Task may seem to be easy, but in fact it isn't... I have a big flow file (>1GB), from which I need to remove, let's say, first line (header) before further processing. So far I had 3 attempt, but none of them works as expected: 1) ReplaceText Works for small files, but the problematic file is too big to load it into memory (I get memory out of bounds exception). 2) SplitText I was trying to use SplitText, but due to this issue I cannot skip the header line in this processor at the moment. In other words - this processor fails whenever Header Line Count > 0. 3) ExecuteProcess I can imagine running a linux command (e.g. tail or sed) to do this job, but it requires saving the flow file to the disk, which might be also costly. Do you have any ideas if this can be done more efficiently? Thanks, Michal
... View more
Labels:
- Labels:
-
Apache NiFi
02-14-2017
02:17 PM
This works! Explained very clearly. Thanks!
... View more
02-13-2017
05:38 PM
@Matt Clarke, no you have not understood my problem - I would like to get a value of a property (e.g. from custom.properties file) called ${nifi.prefix.cat}. nifi.prefix is a literal. The thing is I would like to construct this property name dynamically.
... View more
02-13-2017
05:24 PM
Hi, I have two types of properties: > 1st fixed one ("suffix") and > 2nd "nifi.prefix.[suffix]" - set of defined properties where suffix is a value of 1st property Let's assume that I'd like to set the value of "nifi.filename" dynamically to nifi.prefix.[suffix], where suffix is a value of "suffix". If "suffix" resolves to "cat", "nifi.filename" should be assigned a ${nifi.prefix.cat} property value. Something like below unfortunatelly does not work: nifi.filename = ${nifi.prefix.${suffix}} Do you know if there are any workarounds so that I can achieve what I need? Thanks, Michal
... View more
Labels:
- Labels:
-
Apache NiFi
02-10-2017
06:32 PM
@Pierre Villard OK, thanks a lot for the link once again - the article was very useful, although it's a shame that changes to custom properties are made just as NiFi gets restarted. This limits the use cases for this functionality.
... View more
02-08-2017
03:26 PM
1 Kudo
@ozhurakousky thanks for the tip - I rather need something that I could flexibly change without having to restart NiFi. It looks like the only way to provide properties to ConsumeKafka or GetFile is via system properties. I might need to find some workaround for that.
... View more
02-08-2017
03:21 PM
@Pierre Villard thanks for the tip - that will probably do the job for some of the properties. The problem is that the upstream PG ("A") with all this configuration details is supposed to be re-usable (so that the downstream PG ("B") would remain static - it would just read the parameters). I mean, ideally I would like to have different A's that go to B: A1 -> B, A2 -> B, ... running at the same time. The difference between A1 and A2 would be that A1 has Input Dir = /foo/bar and A2 Input Dir = /brown/fox etc. So in general instances of PG "A" would only be a store of arguments for a call of "B" PG.
... View more