there is a single point of the flow where a queue is filling, and backpressure is being applied: a logattribute to register the ips of the incoming requests Have you tried increasing the "Run Duration" property in the scheduling tab of the LogAttribute processor? Sometimes setting this even at 25ms can considerably increase your flow's throughput. I'd recommend playing around with this setting and seeing what works best. 1.UpdateRecord I could not find a freeform text processor. Here is an example of the incoming data in the form of query parameters: TS=20221108-093700&UID=12345&ETID=22&EGID=44&DID=Hello%20World!&HHID=492489&OID=666 What reader could be used to transform free text to json? If you claim to receive the data in the form of query parameters, perhaps you can use one of the following. For starters, query parameters should be added to your received flowfiles as attributes IF you use the HandleHttpRequest and HandleHttpResponse processors (which ideally you should be doing instead of using the limited ListenHTTP processor). All query parameters received by the HandleHttpRequest processor are added as attributes named http.query.param.<parameter name>. If it is only the query parameters you wish to process in your flow, you can now use the AttributesToJSON processor and list all these query param attributes to generate a json more strictly. If you want to process the query params in addition to your request body, I'd like you to give me a more detailed example of input/output and I'll post a better solution. 2. I didn't 100% understand your use-case. Regarding inserting null values (and not the string 'null'), I found this thread in which a solution is offered. Essentially, you first use your shift operation to get rid of keys for which you want to insert null, then you use a default operation and add null as the default value. I created the following spec and it seems to inject null correctly: [
{
"operation":"shift",
"spec":{
"TS":"TimeStamp",
"UID":"UserID",
"OID":"OpCoID",
"ETID":"EventTypeID",
"EGID":"EventGroupID",
"DID":"DeviceID",
"HHID":{
"0":{
"null":"HouseholdID"
},
" ":{
"null":"HouseholdID"
},
"":{
"null":"HouseholdID"
},
"*":{
"@(2,HID)":"HouseholdID"
}
},
"*":"Custom.&"
}
},
{
"operation":"default",
"spec":{
"Custom":{
},
"HouseholdID": null
}
}
] If you only wish to inject nulls when ETID == 11, the transformation will be a bit more complex.. I think it would be easier as a whole to just extract ETID and HHID to attributes and then use expression language with the "ifElse" operation to evaluate what will be shifted into HouseholdId. 3. I'm afraid I didn't completely understand this point either. If the issue is routing to specific relationships with QueryRecord, every dynamic property you add will create a new relationship. In order for a flowfile to pass to the right relationship, you should add a WHERE clause to your sql query that will only match flowfiles that should pass to that specific route/relationship. I'd be glad to help more 🙂! If you still have questions, providing examples (expected input/output) will make it easier for me to offer a solution for the different parts in your flow. My regards, Eyal.
... View more
