I followed in the instruction from metron wiki installation 4.0 to setup a 3 node cluster. Everything is working fine, however every other day I get one of the node 100% full b/c of the /kafka-log/pcap-0.
Why is this the case? is it because kafka is not processing the messages coming in for pcap topic fast enough?
How can I eliminate this issue in the future? I'm thinking I need to reconfigure or change something in kafka to be more efficient?
Any insight/idea is greatly appreciated.
Are you ingesting pcap data (looks like you are)? This is a high volume data source so you likely need to scale Kafka by adding more nodes/partitions. Are the other nodes filling up too? How many Kafka brokers do you have? A 3 node cluster is really small (especially for something like ingesting pcap) so you're likely undersized anyways.
THank you @rmerriman for your response.
I have 3 Kafka brokers, one on each node. It looks like only one node contains the data for topic pcap and only that particular node is filled up. I modified kafka configuration so that we have three partition for each topic instead of the default of 1. I also modified the retention time and byte size to be small.
How many node roughly would you recommend for a cluster to ingest pcap data?
thanks again for all feedback