How to set Kerberos Kafka + Haproxy (Load Balancer)

[#1]

Hello:

How to use HAProxy to connect for Kafka with Kerberos authentication?

I have three kafka brokers, and i try to use haproxy in front of kafka, but kerberos authenticated failed

My haproxy.conf

listen kafka
 bind *:6677
 mode tcp
 balance roundrobin
 server kafka1 kafka-1.kafka.net:6668 check
 server kafka2 kafka-2.kafka.net:6669 check
 server kafka3 kafka-3.kafka.net:6666 check

I also modified

kafka1 server.properties

• advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668
• listeners=INTERNAL://:6667,LB://:6668
• listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
• inter.broker.listener.name=INTERNAL
• listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.[email protected]"

kafka2 server.properties

• advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6669
• listeners=INTERNAL://:6667,LB://:6669
• listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
• inter.broker.listener.name=INTERNAL
• listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/[email protected]";

kafka3 server.properties

• advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6666
• listeners=INTERNAL://:6667,LB://:6666
• listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
• inter.broker.listener.name=INTERNAL
• listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/[email protected]";

amd use the command 

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --topic my-topic --broker-list gateway.kafka.net:6677 --producer-property security.protocol=SASL_PLAINTEXT

Will get the error:

[2024-10-08 20:07:58,330] ERROR [Producer clientId=console-producer] Connection to node -1 failed authentication due to: Authentication failed due to invalid credentials with SASL mechanism GSSAPI (org.apache.kafka.clients.NetworkClient)
[2024-10-08 20:07:58,330] ERROR Error when sending message to topic my-topic5 with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

[#2]

@polingsky202 
To configure HAProxy to connect three Kafka brokers with Kerberos authentication and resolve the Authentication failed due to invalid credentials with SASL mechanism GSSAPI error, follow these steps:

Step 1: Review the Current Configuration

The provided configuration shows:

1. HAProxy is configured for load balancing using roundrobin.
2. Kafka brokers are set up with:
   • advertised.listeners and listeners for internal and LB connections.
   • SASL GSSAPI configured with Kerberos.

Issue Likely Causes:

• Kerberos principal or keytab file mismatch.
• Improper mapping of advertised listener names.
• Client-side misconfiguration for Kerberos authentication.

Step 2: Correct and Optimize HAProxy Configuration

Update the HAProxy configuration to correctly pass Kerberos authentication to Kafka brokers.

Updated haproxy.cfg

Spoiler

listen kafka
bind *:6677
mode tcp
balance roundrobin
option tcp-check
server kafka1 kafka-1.kafka.net:6668 check
server kafka2 kafka-2.kafka.net:6669 check
server kafka3 kafka-3.kafka.net:6666 check

Key updates above in the haproxy config file:

• Mode TCP: Ensures TCP passthrough for Kerberos authentication.
• Option tcp-check: Validates backend server availability.

Step 3: Verify Kafka Broker Configuration

Ensure the Kerberos configuration for each broker is consistent and properly aligned.

Key Points:

1. advertised.listeners:

   • Ensure the LB listener matches the address clients will connect to via HAProxy (e.g. gateway.kafka.net).

2. Kerberos JAAS Configuration:

   • Validate the listener.name.LB.gssapi.sasl.jaas.config entry for all brokers.
   • Ensure the keyTab file exists and has correct permissions:

Spoiler

ls -l /etc/security/keytabs/kafka.service.keytab

Example Updated kafka1 Broker Configuration:

Spoiler

advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668
listeners=INTERNAL://:6667,LB://:6668 listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT inter.broker.listener.name=INTERNAL listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ doNotPrompt=true useKeyTab=true storeKey=true \ keyTab="/etc/security/keytabs/kafka.service.keytab" \ principal="kafka/[email protected]";

Repeat similar updates for kafka2 and kafka3 with their respective listener ports.

Step 4: Update Kerberos Configuration

Ensure that Kerberos configuration is consistent across all systems.

1. Validate Kerberos krb5.conf ensure the file includes the correct realm and KDC information:

Spoiler

[libdefaults]
default_realm = KAFKA.NET

[realms]
KAFKA.NET = {
kdc = your-kdc-host
admin_server = your-kdc-admin-host
}

2. Test Kerberos Principal: Verify the principal works with the keytab:

Spoiler

kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/[email protected]

Step 5: Verify Client Configuration

The client is attempting to authenticate with Kerberos. Ensure the producer properties are configured correctly updated Producer Command: see below

Spoiler

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh \
--topic my-topic \
--broker-list gateway.kafka.net:6677 \
--producer-property security.protocol=SASL_PLAINTEXT \
--producer-property sasl.kerberos.service.name=kafka

Key Properties:

• security.protocol=SASL_PLAINTEXT: Specifies Kerberos authentication.
• sasl.kerberos.service.name=kafka: Matches the Kerberos principal’s service name.

Step 6: Test and Troubleshoot

Enable Debug Logging: Add -Dsun.security.krb5.debug=true to the JVM options for the client to debug Kerberos issues

Spoiler

export KAFKA_OPTS="-Dsun.security.krb5.debug=true"

Check Logs:

• On the client side, check for detailed Kerberos errors in the output.
• On Kafka brokers, inspect logs for authentication errors:

Spoiler

less /var/log/kafka/server.log

3. Verify Connectivity: Use telnet or nc to confirm connectivity to HAProxy and brokers

Spoiler

telnet gateway.kafka.net 6677
telnet kafka-1.kafka.net 6668

Final Checklist

• Ensure all brokers have consistent Kerberos configurations.
• Verify the client-side security.protocol and sasl.kerberos.service.name settings.
• Ensure HAProxy uses TCP passthrough (mode tcp) for Kerberos.

With these adjustments, the Kerberos authentication error should be resolved. Let me know if further clarification is needed!

Happy hadooping

[#3]

Hello 

@polingsky202 ,

i'm facing the same problem and had the same errors on logs implementing HAproxy with 3 brokers.

Have you solved this issue ?

Thank you for your help.

Best regards.