Member since
08-01-2021
52
Posts
10
Kudos Received
7
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2940 | 11-18-2022 09:06 AM | |
| 4154 | 11-15-2022 05:46 PM | |
| 2973 | 10-12-2022 03:18 AM | |
| 2133 | 10-11-2022 08:52 AM | |
| 5226 | 10-08-2022 08:23 AM |
11-25-2022
09:34 AM
1 Kudo
there is a single point of the flow where a queue is filling, and backpressure is being applied: a logattribute to register the ips of the incoming requests Have you tried increasing the "Run Duration" property in the scheduling tab of the LogAttribute processor? Sometimes setting this even at 25ms can considerably increase your flow's throughput. I'd recommend playing around with this setting and seeing what works best. 1.UpdateRecord I could not find a freeform text processor. Here is an example of the incoming data in the form of query parameters: TS=20221108-093700&UID=12345&ETID=22&EGID=44&DID=Hello%20World!&HHID=492489&OID=666 What reader could be used to transform free text to json? If you claim to receive the data in the form of query parameters, perhaps you can use one of the following. For starters, query parameters should be added to your received flowfiles as attributes IF you use the HandleHttpRequest and HandleHttpResponse processors (which ideally you should be doing instead of using the limited ListenHTTP processor). All query parameters received by the HandleHttpRequest processor are added as attributes named http.query.param.<parameter name>. If it is only the query parameters you wish to process in your flow, you can now use the AttributesToJSON processor and list all these query param attributes to generate a json more strictly. If you want to process the query params in addition to your request body, I'd like you to give me a more detailed example of input/output and I'll post a better solution. 2. I didn't 100% understand your use-case. Regarding inserting null values (and not the string 'null'), I found this thread in which a solution is offered. Essentially, you first use your shift operation to get rid of keys for which you want to insert null, then you use a default operation and add null as the default value. I created the following spec and it seems to inject null correctly: [
{
"operation":"shift",
"spec":{
"TS":"TimeStamp",
"UID":"UserID",
"OID":"OpCoID",
"ETID":"EventTypeID",
"EGID":"EventGroupID",
"DID":"DeviceID",
"HHID":{
"0":{
"null":"HouseholdID"
},
" ":{
"null":"HouseholdID"
},
"":{
"null":"HouseholdID"
},
"*":{
"@(2,HID)":"HouseholdID"
}
},
"*":"Custom.&"
}
},
{
"operation":"default",
"spec":{
"Custom":{
},
"HouseholdID": null
}
}
] If you only wish to inject nulls when ETID == 11, the transformation will be a bit more complex.. I think it would be easier as a whole to just extract ETID and HHID to attributes and then use expression language with the "ifElse" operation to evaluate what will be shifted into HouseholdId. 3. I'm afraid I didn't completely understand this point either. If the issue is routing to specific relationships with QueryRecord, every dynamic property you add will create a new relationship. In order for a flowfile to pass to the right relationship, you should add a WHERE clause to your sql query that will only match flowfiles that should pass to that specific route/relationship. I'd be glad to help more 🙂! If you still have questions, providing examples (expected input/output) will make it easier for me to offer a solution for the different parts in your flow. My regards, Eyal.
... View more
11-25-2022
07:43 AM
Hi @ripo , I've tried it out in my own environment and it does not seem like ports have an 'ENABLED' state. If a port is DISABLED, you can only update it to be STOPPED. from a STOPPED state, you can either switch the port to RUNNING or back to DISABLED.
... View more
11-25-2022
07:25 AM
Hey @F_Amini , If the data has already been loaded into nifi, I don't believe you can do much other than to increase your resources. I see you're already familar with increasing concurrent tasks (it appears your HashContent processor has an increased number) - perhaps if your entire flow is getting bottlenecked by a single processor, you could temporarily increase its tasks even further (at the cost of other processors getting less).
... View more
11-25-2022
05:11 AM
Hi @Fredi , The processor 'ForkRecord' is exactly what you are looking for. However, I am currently trying to configure it to split exactly as you have described and am having trouble getting it to work... I am attempting to use the 'extract' mode, with a fork path of "/Key1[*]/NestedKey2" which, according to the documentation, is supposed to do exactly what you described (split on nested json). For some reason the output is coming out empty though. Perhaps someone more familiar with the processor could reply and explain how to use it correctly for your use-case.
... View more
11-19-2022
01:35 PM
I am bumping this question in hopes someone might know of a better solution
... View more
11-18-2022
09:06 AM
After testing, in order to remove a variable you need to send an update request where that variable's value is null.
... View more
11-18-2022
08:25 AM
A couple of reasons: At the base level, operations in the UI all use the REST API - if you use chrome's devtools and go to the networking tab, you can see the exact REST API route used for every click you do on the canvas. If you'd ever want to automate something you know you can manually do in the UI, using this trick you can perfectly replicate it with with RESTful calls. In the same vain, the REST API has far more routes/operations available. The CLI tools seems to have around ~70 operations, whilst the rest api seems to have over 200. The CLI commands are part of the nifi toolkit. Whilst the toolkit does get updated, I believe it does not get the same level of attention as nifi's main codebase and as such it'd be better not to rely on it completely. This is not to say you can't use the CLI tool - rather, just my opinion on the matter and some insight for how my team writes automations on nifi 🙂
... View more
11-18-2022
12:02 AM
Hey @syntax_ , Here is the documentation on NIFI's toolkit CLI commands. It seems there are commands for "pg-stop" and "pg-start", which I believe will allow you to start/stop and manage specific flows inside your canvas by using their IDs. Personally, I would recommend you use NIFI's REST API for automations rather than CLI commands. Hope this helps 🙂
... View more
11-15-2022
06:11 PM
Hello Miguel, There are probably ways to optimize tha amount of processors you use, but as a start I would try and monitor where the slowness comes from. Is there any backpressure in the flow (files queueing up before a processor)? Are you sending requests at the same rate? if so does that mean those 6000 records/second you lost translate to 5xx responses from nifi to the origin service? In terms of optimizing, perhaps you can transform null values with your initial JoltTransformJson. it sounds like you use a lot of ReplaceText processors too - perhaps UpdateRecord would be of more use for updating your fields. I'd recommend looking at QueryRecord for your validation/routing instead of using UpdateAttribute -> RouteOnAttribute (if applicable and you route based on the json fields).
... View more
11-15-2022
05:56 PM
Hello Mahendra, Have you tried sending an update request that includes all the variables except the one you wish to delete? If that doesn't work I'll run some tests next time I'm at office and post an update.
... View more