Is there any specific reason why you need a load balancer? Kafka is supposed to work without a separated load balancer, and handle the load on the cluster. With the load balancer the client might fail to connect when the broker tries to redirect the connections to another listener.
The brokers section in the config for a client is used to try getting the actual connect parameter (listeners) from the cluster. You typically provide multiple brokers to avoid having an issue if the initial broker is down while connecting.
When you try to change the listener configs in the kafka brokers (or zookeeper), i think you actually disable the cluster, as the broker are communicating to each other as well.