Kafka KRaft Cluster SASL Inter-Broker Authentication Failure: Unexpected Request During Handshake

I am setting up a 3-node Apache Kafka cluster using KRaft mode (3.9.0) with SCRAM-SHA-256 for inter-broker communication on the SASL_PLAINTEXT listener. The cluster is unable to form a quorum due to persistent SASL authentication failures.

All configuration files and passwords have been synchronized. The exact error points to a handshake failure, but the settings appear correct.

Environment and Node Details Kafka Version: 3.9.0 (KRaft mode)

Security Protocol: SASL_PLAINTEXT (SCRAM-SHA-256)

JAAS File Location: /opt/kafka/config/kafka_server_jaas.conf

Nodes:

Node 1: 172.20.24.155 (node.id=1)

Node 2: 172.20.24.156 (node.id=2)

Node 3: 172.20.24.157 (node.id=3)

Errors The primary error is a SASL handshake failure, leading the client (connecting broker) to prematurely send an API request. This confirms that the broker is attempting to authenticate but is immediately rejected by the remote server.

NODE 1 Logs (Example of persistent failures after configuration changes)

Nov 13 13:00:10 NDCPRDKAFKA01 kafka-server-start.sh[3995292]: [2025-11-13 13:00:10,518] INFO [SocketServer listenerType=BROKER, nodeId=1] Failed authentication with /172.20.24.156 (channelId=172.20.24.155:9092-172.20.24.156:51726-3) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)



NODE 2 Logs (Similar error, often reporting disconnection issues for itself and other nodes)

Nov 13 13:00:19 NDCPRDKAFKA02 kafka-server-start.sh[2649709]: [2025-11-13 13:00:19,258] INFO [SocketServer listenerType=BROKER, nodeId=2] Failed authentication with /172.20.24.157 (channelId=172.20.24.156:9092-172.20.24.157:54096-35) (Unexpected Kafka request of type DESCRIBE_CLUSTER during SASL handshake.) (org.apache.kafka.common.network.Selector)



NOTE: Previous error (now fixed) was:

java.lang.IllegalArgumentException: Could not find a 'KafkaServer'... entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set

Configuration Files (Current State)

1. server.properties (All Nodes) The core security settings are configured identically on all nodes, differing only by node.id and local IP address in the listeners block.

process.roles=broker,controller
node.id=<1, 2, or 3>
controller.quorum.voters=1@172.20.24.155:9093,2@172.20.24.156:9093,3@172.20.24.157:9093

# Network Configuration
listeners=SASL_PLAINTEXT://172.20.24.<155/156/157>:9092,CONTROLLER://172.20.24.<155/156/157>:9093
advertised.listeners=SASL_PLAINTEXT://172.20.24.<155/156/157>:9092
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# SASL Security Configuration
inter.broker.listener.name=SASL_PLAINTEXT
sasl.enabled.mechanisms=SCRAM-SHA-256
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
sasl.mechanism.controller.protocol=SCRAM-SHA-256

2. kafka_server_jaas.conf (All Nodes) The JAAS file is an external file, and the credentials are set as follows (based on the user/password history provided):

Node,User,Password
Node 1,broker1,secret1
Node 2,broker2,secret2
Node 3,broker3,secret3

Example Content (Node 1):

KafkaServer {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="broker1"
password="secret1";
};

KafkaClient {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="broker1"
password="secret1";
};

3. kafka.service (Systemd Loader - All Nodes) The environment variable loading the external JAAS file is now correctly configured, resolving the previous IllegalArgumentException.

[Service]
User=kafka
Group=kafka
# ... other properties
Environment="KAFKA_OPTS=-Djava.security.manager=allow -Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
# ...

4. Users Created in KRaft Metadata The broker principals were created and synchronized using the following command pattern (running against the working listener):

# Example for broker1
/opt/kafka/bin/kafka-configs.sh --bootstrap-server 172.20.24.155:9092 --alter --add-config 'SCRAM-SHA-256=[password=secret1]' --entity-type users --entity-name broker1

# Similar commands were run for broker2 and broker3 with their respective passwords.

This confirms the hashed password in the KRaft metadata should match the plaintext password in the JAAS file.

Given that the following checks are complete:

1. SASL is enabled in server.properties.
2. JAAS is loaded via systemd (KAFKA_OPTS).
3. Plaintext passwords in the JAAS file are identical to the passwords used to create the SCRAM users in the KRaft metadata.

Why is the inter-broker SASL handshake still failing, resulting in the "Unexpected Kafka request of type DESCRIBE_CLUSTER during SASL handshake" error?

Is there a configuration detail I am missing, such as a requirement for the CONTROLLER listener security, or an issue related to the SCRAM negotiation when the cluster is initially forming its quorum?