Support Questions

Find answers, ask questions, and share your expertise

How to set Kerberos Kafka + Haproxy (Load Balancer)

avatar

Hello:

How to use HAProxy to connect for Kafka with Kerberos authentication?

I have three kafka brokers, and i try to use haproxy in front of kafka, but kerberos authenticated failed

My haproxy.conf

listen kafka
 bind *:6677
 mode tcp
 balance roundrobin
 server kafka1 kafka-1.kafka.net:6668 check
 server kafka2 kafka-2.kafka.net:6669 check
 server kafka3 kafka-3.kafka.net:6666 check

I also modified

kafka1 server.properties

  • advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668
  • listeners=INTERNAL://:6667,LB://:6668
  • listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
  • inter.broker.listener.name=INTERNAL
  • listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET"

kafka2 server.properties

  • advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6669
  • listeners=INTERNAL://:6667,LB://:6669
  • listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
  • inter.broker.listener.name=INTERNAL
  • listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET";

kafka3 server.properties

  • advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6666
  • listeners=INTERNAL://:6667,LB://:6666
  • listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
  • inter.broker.listener.name=INTERNAL
  • listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET";

amd use the command 

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --topic my-topic --broker-list gateway.kafka.net:6677 --producer-property security.protocol=SASL_PLAINTEXT

 

Will get the error:

[2024-10-08 20:07:58,330] ERROR [Producer clientId=console-producer] Connection to node -1 failed authentication due to: Authentication failed due to invalid credentials with SASL mechanism GSSAPI (org.apache.kafka.clients.NetworkClient)
[2024-10-08 20:07:58,330] ERROR Error when sending message to topic my-topic5 with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

2 REPLIES 2

avatar
Master Mentor

@polingsky202 
To configure HAProxy to connect three Kafka brokers with Kerberos authentication and resolve the Authentication failed due to invalid credentials with SASL mechanism GSSAPI error, follow these steps:

Step 1: Review the Current Configuration

The provided configuration shows:

  1. HAProxy is configured for load balancing using roundrobin.
  2. Kafka brokers are set up with:
    • advertised.listeners and listeners for internal and LB connections.
    • SASL GSSAPI configured with Kerberos.

Issue Likely Causes:

  • Kerberos principal or keytab file mismatch.
  • Improper mapping of advertised listener names.
  • Client-side misconfiguration for Kerberos authentication.

Step 2: Correct and Optimize HAProxy Configuration

Update the HAProxy configuration to correctly pass Kerberos authentication to Kafka brokers.

Updated haproxy.cfg

Spoiler
listen kafka
bind *:6677
mode tcp
balance roundrobin
option tcp-check
server kafka1 kafka-1.kafka.net:6668 check
server kafka2 kafka-2.kafka.net:6669 check
server kafka3 kafka-3.kafka.net:6666 check

Key updates above in the haproxy config file:

  • Mode TCP: Ensures TCP passthrough for Kerberos authentication.
  • Option tcp-check: Validates backend server availability.

Step 3: Verify Kafka Broker Configuration

Ensure the Kerberos configuration for each broker is consistent and properly aligned.

Key Points:

  1. advertised.listeners:

    • Ensure the LB listener matches the address clients will connect to via HAProxy (e.g. gateway.kafka.net).
  2. Kerberos JAAS Configuration:

    • Validate the listener.name.LB.gssapi.sasl.jaas.config entry for all brokers.
    • Ensure the keyTab file exists and has correct permissions:
Spoiler
ls -l /etc/security/keytabs/kafka.service.keytab

Example Updated kafka1 Broker Configuration:

Spoiler
advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668
listeners=INTERNAL://:6667,LB://:6668 listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT inter.broker.listener.name=INTERNAL listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ doNotPrompt=true useKeyTab=true storeKey=true \ keyTab="/etc/security/keytabs/kafka.service.keytab" \ principal="kafka/gateway.kafka.net@KAFKA.NET";

Repeat similar updates for kafka2 and kafka3 with their respective listener ports.

Step 4: Update Kerberos Configuration

Ensure that Kerberos configuration is consistent across all systems.

  1. Validate Kerberos krb5.conf ensure the file includes the correct realm and KDC information:

Spoiler

[libdefaults]
default_realm = KAFKA.NET

[realms]
KAFKA.NET = {
kdc = your-kdc-host
admin_server = your-kdc-admin-host
}

2. Test Kerberos Principal: Verify the principal works with the keytab:

Spoiler
kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/gateway.kafka.net@KAFKA.NET

Step 5: Verify Client Configuration

The client is attempting to authenticate with Kerberos. Ensure the producer properties are configured correctly updated Producer Command: see below

Spoiler
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh \
--topic my-topic \
--broker-list gateway.kafka.net:6677 \
--producer-property security.protocol=SASL_PLAINTEXT \
--producer-property sasl.kerberos.service.name=kafka

Key Properties:

  • security.protocol=SASL_PLAINTEXT: Specifies Kerberos authentication.
  • sasl.kerberos.service.name=kafka: Matches the Kerberos principal’s service name.

Step 6: Test and Troubleshoot

Enable Debug Logging: Add -Dsun.security.krb5.debug=true to the JVM options for the client to debug Kerberos issues

Spoiler
export KAFKA_OPTS="-Dsun.security.krb5.debug=true"

Check Logs:

  • On the client side, check for detailed Kerberos errors in the output.
  • On Kafka brokers, inspect logs for authentication errors:
Spoiler
less /var/log/kafka/server.log

3. Verify Connectivity: Use telnet or nc to confirm connectivity to HAProxy and brokers

Spoiler
telnet gateway.kafka.net 6677
telnet kafka-1.kafka.net 6668

Final Checklist

  • Ensure all brokers have consistent Kerberos configurations.
  • Verify the client-side security.protocol and sasl.kerberos.service.name settings.
  • Ensure HAProxy uses TCP passthrough (mode tcp) for Kerberos.

With these adjustments, the Kerberos authentication error should be resolved. Let me know if further clarification is needed!

Happy hadooping









avatar
New Contributor

Hello 

 
i'm facing the same problem and had the same errors on logs implementing HAproxy with 3 brokers.
Have you solved this issue ?
 
Thank you for your help.
 
Best regards.