Support Questions

Find answers, ask questions, and share your expertise

KafkaSpout Issue with 1 worker node and 1 worker process



I am trying to setup a baseline for performance of our topology. I started with 1 worker node and 1 worker processes

We have 1 Kafka Spout, 1 Processing Bolt and 1 HdfsBolt. I set the parallelism for all of them to 1.

Below are other storm configs. We are using 1 D14_V2 in Azure for our worker/supervisor node. 2 D4 v2 for Nimbus and 3 D12 v2 for Zookeeprs

  • worker.childopts: -Xmx4000m-Xms4000m (4GB memory)
  • log level: WARN
  • supervisor.childopts: -Xmx1024m _JAAS_PLACEHOLDER{{jmxremote_port}} -javaagent:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM -Detwlogger.component=supervisor
  • topology.max.spout.pending: 250
  • topology.worker.max.heap.size.mb: 4000
  • topology.acker.executors: 6
  • topology.workers: 1

I deployed the topology to the Storm cluster and pumped messages into Kafka topic. But KafkaSpout doesn't consume any of those messages. I did check to make sure the topic has messages.

In our KafkaSpout, we give a static list of broker nodes list. I tried telnet from the worker node to both Kafka brokers and zookeepers and both of them were successful in getting connections.

Does anyone have any insight into what could be wrong with KafkaSpout thats preventing it from consuming messages from kafka?


Here's a snippet from a working scenario similar to yours that works just fine for me.

TopologyBuilder builder = new TopologyBuilder();
BrokerHosts hosts = new ZkHosts("zk1:2181,zk2:2181,zk3:2181");
SpoutConfig sc = new SpoutConfig(hosts,
        "s20-logs", "/s20-logs",
sc.scheme = new SchemeAsMultiScheme(new StringScheme());
KafkaSpout spout = new KafkaSpout(sc);
builder.setSpout("log-spout", spout, 1);

If you notice, you'll see that my BrokerHosts are really the list of ZooKeeper instances. I'm running this on a HDP 2.5 cluster which is Storm 1.0.1 and I constructed this ZkHosts class after reviewing the notes in

Might be worth a try for you. Either way, good luck and Happy Hadooping/Storming!


Were you able to resolve your issue ??

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.