No Workers in Storm for Squid Topology

[#1]

I'm attempting to follow the new data source tutorial for Metron. I have squid up and running, the topology was added to Storm, the topic added to kafka, NiFi is setup and everything seems to be correct, but it is not working.

When I look at the Storm UI, there are no workers running for the Squid topology. I'd assume I'd see one, since the config states 1. There are entries in the /data1/kafka-log/squid-0/ directory for the log files, but that's the only place I see any related data from the squidclient entries.

Any idea where I can start looking?

Possibly related question, after I installed Metron I got a handful of records in the UI. Since then, nothing is being shown. The system has been up for an hour and it's only displaying ~15 minutes of data?

[#2 · Accepted solution]

I figured this out. It was due to the configuration of the Storm cluster. I didn't realize the cluster had a max of 4 workers set and each worker had already been committed to a topology.

[#3]

@Rob Becker How did you resolve this issue? Someone suggested either deactivating or killing another of the topologies to free up a worker. I tried both and neither works. Did you add another port? If so, how? Thanks for your help.

[#4]

@Rita McKissick I ended up removing one of the other topologies. I think I ran "storm kill <topology>". My implementation of Metron is only going to take advantage of the enrichment topology. I removed bro and the other ones that I'm not going to use.

[#5]

You can also add an additional storm worker in Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning an additional port to the list

[#6]

Yes KC. Adding an extra port for squid has worked for me . Thanks.