Storm complete latency more than individual bolts

[#1]

We have a storm topology that usually runs fine. We have no errors in the logs and times are fast. However, sometimes we get spikes in the "complete latency"

Here are the individual bolts:

So my question is: What causes the Complete latency to be much more than the individual bolts? ( We are using the Microsoft spout and event hubs if that helps).

[#2 · Accepted solution]

Latencies for each component excludes queue wait time and transfer latency between worker. 'complete latency' means all the nodes in tuple tree are acked, so it reflects slowest path of the tree.

Btw, behind the scene, 'complete latency' includes waiting time for spout to handle ack from acker, and if your spout spends long time in nextTuple it will heavily affect 'complete latency'. It is fixed from STORM-1742 and will be included to next release. (Storm 1.0.3 for Apache side, but not sure which HDP / HDF versions contain this fix. Please let me know your HDP/HDF version.)

Hope this helps.