Support Questions
Find answers, ask questions, and share your expertise

Spout parallism is not working for storm 1.1.0

Hello,

We're consuming data from Solace queue(via JMS APIs) and we start the topology with 3 executors(and same tasks) for this spout.And problem we're getting is that all data is being consumed from one executor only and other executors are idle which makes the overall consumption very slow and after consuming few hundred thousand records this executor breaks with out of memory error.

This was working fine with earlier version of HDP& Storm(1.0.1-2.5.3).

The version we're using now ;Storm version : 1.1.0, HDP version : 2.6.1

So, we 're looking for solution of starting all executors at same time to distribute the load?

2 REPLIES 2

Re: Spout parallism is not working for storm 1.1.0

Rising Star

@Dhiraj Sardana

Could you please share your storm UI screenshot as well capturing the spout and bolts metrics.

Thank you.

Re: Spout parallism is not working for storm 1.1.0

PFA screenshot from Spout metrics from Storm UI . : storm-spout-executors.png

And we have set distributed to 'true' while creating spout. So distribution(multithreading) is enabled.

Also, though we have fixed some issues in bolts and overall performance is good, but we might have more load so if if spout level as well parrallism can work , that would be great.

It might be that Solace(queue provider) does not supports multithreading?