We have a dev cluster with 3 nodes and we have installed Kafka on them. However during ingestion from Mysql binlog, we notice that 2 of the nodes are a slower, between the 2 nodes, one node has the lowest byte out to consumer. We can see that in the grafana graph.
The specs of the servers are as follow:
20cores cpu, 140GB memory 900GB RAID 5 HDD
12 cores cpu, 94GB memory 900GB RAID0 HDD
12 cores cpu,
94GB memory 900GB RAID0 HDD
No matter how many partitions we set, the slave node2 will always be the slowest, I also notice there are some bad sector in the harddisk of slave node2, will it affect the speed of Kafka?
Hope someone with experience can shed some lights on this.