@polingsky202 To configure HAProxy to connect three Kafka brokers with Kerberos authentication and resolve the Authentication failed due to invalid credentials with SASL mechanism GSSAPI error, follow these steps:
Step 1: Review the Current Configuration The provided configuration shows:
HAProxy is configured for load balancing using roundrobin .Kafka brokers are set up with:advertised.listeners and listeners for internal and LB connections.SASL GSSAPI configured with Kerberos. Issue Likely Causes :
Kerberos principal or keytab file mismatch. Improper mapping of advertised listener names. Client-side misconfiguration for Kerberos authentication. Step 2: Correct and Optimize HAProxy Configuration Update the HAProxy configuration to correctly pass Kerberos authentication to Kafka brokers.
Updated haproxy.cfg Spoiler (Highlight to read) listen kafka bind *:6677 mode tcp balance roundrobin option tcp-check server kafka1 kafka-1.kafka.net:6668 check server kafka2 kafka-2.kafka.net:6669 check server kafka3 kafka-3.kafka.net:6666 check
listen kafkabind *:6677mode tcpbalance roundrobinoption tcp-checkserver kafka1 kafka-1.kafka.net:6668 checkserver kafka2 kafka-2.kafka.net:6669 checkserver kafka3 kafka-3.kafka.net:6666 check
Key updates above in the haproxy config file :Mode TCP : Ensures TCP passthrough for Kerberos authentication.Option tcp-check : Validates backend server availability.Step 3: Verify Kafka Broker Configuration Ensure the Kerberos configuration for each broker is consistent and properly aligned.
Key Points: advertised.listeners :
Ensure the LB listener matches the address clients will connect to via HAProxy (e.g. gateway.kafka.net ). Kerberos JAAS Configuration :
Validate the listener.name.LB.gssapi.sasl.jaas.config entry for all brokers. Ensure the keyTab file exists and has correct permissions: Spoiler (Highlight to read) ls -l /etc/security/keytabs/kafka.service.keytab
ls -l /etc/security/keytabs/kafka.service.keytab
Example Updated kafka1 Broker Configuration: Spoiler (Highlight to read) advertised.listeners =INTERNAL://:6667 ,LB://gateway.kafka.net:6668 listeners =INTERNAL://:6667 ,LB://:6668 listener.security.protocol.map =INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT inter.broker.listener.name =INTERNAL listener.name.LB.gssapi.sasl.jaas.config =com.sun.security.auth.module.Krb5LoginModule required \ doNotPrompt =true useKeyTab=true storeKey=true \ keyTab ="/etc/security/keytabs/kafka.service.keytab" \ principal ="kafka/gateway.kafka.net@KAFKA.NET"
advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668listeners=INTERNAL://:6667,LB://:6668 listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT inter.broker.listener.name=INTERNAL listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ doNotPrompt=true useKeyTab=true storeKey=true \ keyTab="/etc/security/keytabs/kafka.service.keytab" \ principal="kafka/gateway.kafka.net@KAFKA.NET";
Repeat similar updates for kafka2 and kafka3 with their respective listener ports.
Step 4: Update Kerberos Configuration Ensure that Kerberos configuration is consistent across all systems.
Validate Kerberos krb5.conf e nsure the file includes the correct realm and KDC information:
Spoiler (Highlight to read) [libdefaults] default_realm = KAFKA.NET
[realms] KAFKA.NET = { kdc = your-kdc-host admin_server = your-kdc-admin-host }
[libdefaults]default_realm = KAFKA.NET[realms]KAFKA.NET = {kdc = your-kdc-hostadmin_server = your-kdc-admin-host}
2. Test Kerberos Principal : Verify the principal works with the keytab:Spoiler (Highlight to read) kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/gateway.kafka.net@KAFKA.NET
kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/gateway.kafka.net@KAFKA.NET
Step 5: Verify Client Configuration The client is attempting to authenticate with Kerberos. Ensure the producer properties are configured correctly u pdated Producer Command: see below
Spoiler (Highlight to read) /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh \ --topic my-topic \ --broker-list gateway.kafka.net:6677 \ --producer-property security.protocol=SASL_PLAINTEXT \ --producer-property sasl.kerberos.service.name=kafka
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh \--topic my-topic \--broker-list gateway.kafka.net:6677 \--producer-property security.protocol=SASL_PLAINTEXT \--producer-property sasl.kerberos.service.name=kafka
Key Properties :security.protocol=SASL_PLAINTEXT : Specifies Kerberos authentication.sasl.kerberos.service.name=kafka: Matches the Kerberos principal’s service name.Step 6: Test and Troubleshoot Enable Debug Logging : Add -Dsun.security.krb5.debug=true to the JVM options for the client to debug Kerberos issues
Spoiler (Highlight to read) export KAFKA_OPTS="-Dsun.security.krb5.debug=true"
export KAFKA_OPTS="-Dsun.security.krb5.debug=true"
Check Logs :
On the client side, check for detailed Kerberos errors in the output. On Kafka brokers, inspect logs for authentication errors: Spoiler (Highlight to read) less /var/log/kafka/server.log
less /var/log/kafka/server.log
3. Verify Connectivity : Use telnet or nc to confirm connectivity to HAProxy and brokers Spoiler (Highlight to read) telnet gateway.kafka.net 6677 telnet kafka-1.kafka.net 6668
telnet gateway.kafka.net 6677telnet kafka-1.kafka.net 6668
Final Checklist Ensure all brokers have consistent Kerberos configurations. Verify the client-side security.protocol and sasl.kerberos.service.name settings. Ensure HAProxy uses TCP passthrough (mode tcp ) for Kerberos. With these adjustments, the Kerberos authentication error should be resolved. Let me know if further clarification is needed!
