I have a question regarding Storm. I have devices which are sending data about their actual state (running or not), their location and so on in realtime. Further I have data about the energy consumption of each device in a database. What I would like to do is to join the information of the energy consumption with the device data to compute the actual energy consumption, e.g. in a particular location.
My question: would it be correct to use Kafka to get the data stream from the devices and to use a Storm Bolt to join the data and to compute the energy consumption?
Thank you very much in advance.
Can you qualify the 'device' a bit more? How small/large is it? Is it a server? A microcontroller performing analog readings in the field? Battery operated? Do you expect to be able to install a Kafka client on this client?
Enriching an event is one of the key reasons CEP tools like storm are necessary. Data coming from devices often does not have all of the details necessary to obtain the potential value from the event. It is often necessary to enrich the event by adding data from some data store. Make sure to load the connection to the store on prepare and to properly size the data store to ensure it will be able to support the required TPS. In general, any kind of store can be used but traditional RDBMS will often be overwhelmed by the volume generated by IOT processing platforms. Key value stores like HBase and if possible, in-memory stores will generally work best for this purpose.