Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Storm complete latency more than individual bolts

avatar
Super Guru

We have a storm topology that usually runs fine. We have no errors in the logs and times are fast. However, sometimes we get spikes in the "complete latency"

enter image description here

Here are the individual bolts:

enter image description here

So my question is: What causes the Complete latency to be much more than the individual bolts? ( We are using the Microsoft spout and event hubs if that helps).

1 ACCEPTED SOLUTION

avatar
Contributor

Latencies for each component excludes queue wait time and transfer latency between worker. 'complete latency' means all the nodes in tuple tree are acked, so it reflects slowest path of the tree.

Btw, behind the scene, 'complete latency' includes waiting time for spout to handle ack from acker, and if your spout spends long time in nextTuple it will heavily affect 'complete latency'. It is fixed from STORM-1742 and will be included to next release. (Storm 1.0.3 for Apache side, but not sure which HDP / HDF versions contain this fix. Please let me know your HDP/HDF version.)

Hope this helps.

View solution in original post

1 REPLY 1

avatar
Contributor

Latencies for each component excludes queue wait time and transfer latency between worker. 'complete latency' means all the nodes in tuple tree are acked, so it reflects slowest path of the tree.

Btw, behind the scene, 'complete latency' includes waiting time for spout to handle ack from acker, and if your spout spends long time in nextTuple it will heavily affect 'complete latency'. It is fixed from STORM-1742 and will be included to next release. (Storm 1.0.3 for Apache side, but not sure which HDP / HDF versions contain this fix. Please let me know your HDP/HDF version.)

Hope this helps.