I have a Storm topology running in a distributed environment across 5 linux nodes.
i have one nimbus and 4 supervisors. I have a Kafka Spout that receives a message and then forwards it into a ParseBolt that will parse the raw message and send it to a prediction bolt then to a hbase bolt. the prediction bolt is a python 2.7 bolt.
my major problem is that this topology takes lot of time to execute prediction bolt and the latency reach 30 minutes sometimes. i have:
6 supervisor ports
40 parallelism_hint (for prediction bolt)
40 num task (for prediction bolt)
how can i reduce latency and reach a real time prediction bolt.