Member since
03-22-2019
3
Posts
0
Kudos Received
0
Solutions
07-12-2019
03:20 AM
Hi, I'm trying to create a flow that can received multiple separate messages that each of part of a "complete" record. Once all the parts of a given record are received, then processes that record. The parts of the record can be received out of order and intermixed together. Example data (2 parts to data in example. Some records have more then 2 parts): location_record
{
measurement_id: 123,
measurement_location: /foo/
} file_record
{
measurement_id: 123,
measurement_file: bar.dat
} Each record is received on a different input. The order of which comes first is not guaranteed. Multiple location or file records may be received before the other part of a given id is received. My thought right now is to have a custom service that maintains a map of <id,parts[pair(part_label,part_value)]>. Then have custom processors: 1) Put- when flow file is received from each input, add given attribute (measurement_file, measurement_location, etc) into map using id as key. attribute to store and attribute for key would be properties. Access to map is provided via service api 2) Get- Loop over map at a given interval and look for any id that has required number of pieces (2 in this example). For each id that has all pieces, create file flow and transfer for processing. I'd like to have the storage is something beyond just memory so it can persist. Is there a way to do this with existing processors/services/cachemaps? Any thoughts, comments or suggestions for better approaches are greatly appreciated. -Aaron
... View more
Labels:
03-22-2019
03:51 PM
Hi Gergely, This short version of my question: Can I set the Run Schedule value dynamically? I have an array of different sensors that need to be polled and have value saved/reported. This would be a sample flow: I need to be able to change the interval of each sensor pulling. I'm thinking the Monitor Controller being able to receive a json message such as: { "Sensor1": 60, "Sensor2": 600, "Sensor3": 600, } To set Sensor1 to 60s interval, Sensor 2 and 3 to 10 minutes. Then send in: { "Sensor1": 600, "Sensor2": 60, "Sensor3": 60, } To change Sensor 2 and 3 to 60s and Sensor 1 to 10 minutes when needed. I need the flow to be autonomous beyond the control messages. I can't have an external event "poke" the pollers constantly. Don't know if there is a way to do this in NiFi or if I have to implement my own bookkeeping in a custom processor. Thanks.
... View more
03-22-2019
05:24 AM
Hello, I am trying to set up a processor that runs at a variable interval. I'd like to be able to change that interval amount dynamically by reading the content/setting an attribute from control message received by the flow. Is that possible? I have looked but not been able to find a way to do that. Thank you. -Aaron
... View more
Labels: