question How to set Kerberos Kafka + Haproxy (Load Balancer) in Support Questions

How to set Kerberos Kafka + Haproxy (Load Balancer)

polingsky202 — Tue, 08 Oct 2024 05:12:47 GMT

Hello:

How to use HAProxy to connect for Kafka with Kerberos authentication?

I have three kafka brokers, and i try to use haproxy in front of kafka, but kerberos authenticated failed

My haproxy.conf

listen kafka
bind *:6677
mode tcp
balance roundrobin
server kafka1 kafka-1.kafka.net:6668 check
server kafka2 kafka-2.kafka.net:6669 check
server kafka3 kafka-3.kafka.net:6666 check

I also modified

kafka1 server.properties

advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668
listeners=INTERNAL://:6667,LB://:6668
listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
inter.broker.listener.name=INTERNAL
listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET"

kafka2 server.properties

advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6669
listeners=INTERNAL://:6667,LB://:6669
listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
inter.broker.listener.name=INTERNAL
listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET";

kafka3 server.properties

advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6666
listeners=INTERNAL://:6667,LB://:6666
listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
inter.broker.listener.name=INTERNAL
listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET";

amd use the command

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --topic my-topic --broker-list gateway.kafka.net:6677 --producer-property security.protocol=SASL_PLAINTEXT

Will get the error:

[2024-10-08 20:07:58,330] ERROR [Producer clientId=console-producer] Connection to node -1 failed authentication due to: Authentication failed due to invalid credentials with SASL mechanism GSSAPI (org.apache.kafka.clients.NetworkClient)
[2024-10-08 20:07:58,330] ERROR Error when sending message to topic my-topic5 with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

Re: How to set Kerberos Kafka + Haproxy (Load Balancer)

Shelton — Sun, 22 Dec 2024 13:33:21 GMT

@polingsky202
To configure HAProxy to connect three Kafka brokers with Kerberos authentication and resolve the Authentication failed due to invalid credentials with SASL mechanism GSSAPI error, follow these steps:

Step 1: Review the Current Configuration

The provided configuration shows:

HAProxy is configured for load balancing using roundrobin.
Kafka brokers are set up with:
- advertised.listeners and listeners for internal and LB connections.
- SASL GSSAPI configured with Kerberos.

Issue Likely Causes:

Kerberos principal or keytab file mismatch.
Improper mapping of advertised listener names.
Client-side misconfiguration for Kerberos authentication.

Step 2: Correct and Optimize HAProxy Configuration

Update the HAProxy configuration to correctly pass Kerberos authentication to Kafka brokers.

Updated haproxy.cfg

listen kafka
bind *:6677
mode tcp
balance roundrobin
option tcp-check
server kafka1 kafka-1.kafka.net:6668 check
server kafka2 kafka-2.kafka.net:6669 check
server kafka3 kafka-3.kafka.net:6666 check

Key updates above in the haproxy config file:

Mode TCP: Ensures TCP passthrough for Kerberos authentication.
Option tcp-check: Validates backend server availability.

Step 3: Verify Kafka Broker Configuration

Ensure the Kerberos configuration for each broker is consistent and properly aligned.

Key Points:

advertised.listeners:
- Ensure the LB listener matches the address clients will connect to via HAProxy (e.g. gateway.kafka.net).
Kerberos JAAS Configuration:
- Validate the listener.name.LB.gssapi.sasl.jaas.config entry for all brokers.
- Ensure the keyTab file exists and has correct permissions:

ls -l /etc/security/keytabs/kafka.service.keytab

Example Updated kafka1 Broker Configuration:

advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668
listeners=INTERNAL://:6667,LB://:6668 listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT inter.broker.listener.name=INTERNAL listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ doNotPrompt=true useKeyTab=true storeKey=true \ keyTab="/etc/security/keytabs/kafka.service.keytab" \ principal="kafka/gateway.kafka.net@KAFKA.NET";

Repeat similar updates for kafka2 and kafka3 with their respective listener ports.

Step 4: Update Kerberos Configuration

Ensure that Kerberos configuration is consistent across all systems.

Validate Kerberos krb5.conf ensure the file includes the correct realm and KDC information:

[libdefaults]
default_realm = KAFKA.NET

[realms]
KAFKA.NET = {
kdc = your-kdc-host
admin_server = your-kdc-admin-host
}

2. Test Kerberos Principal: Verify the principal works with the keytab:

kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/gateway.kafka.net@KAFKA.NET

Step 5: Verify Client Configuration

The client is attempting to authenticate with Kerberos. Ensure the producer properties are configured correctly updated Producer Command: see below

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh \
--topic my-topic \
--broker-list gateway.kafka.net:6677 \
--producer-property security.protocol=SASL_PLAINTEXT \
--producer-property sasl.kerberos.service.name=kafka

Key Properties:

security.protocol=SASL_PLAINTEXT: Specifies Kerberos authentication.
sasl.kerberos.service.name=kafka: Matches the Kerberos principal’s service name.

Step 6: Test and Troubleshoot

Enable Debug Logging: Add -Dsun.security.krb5.debug=true to the JVM options for the client to debug Kerberos issues

export KAFKA_OPTS="-Dsun.security.krb5.debug=true"

Check Logs:

On the client side, check for detailed Kerberos errors in the output.
On Kafka brokers, inspect logs for authentication errors:

less /var/log/kafka/server.log

3. Verify Connectivity: Use telnet or nc to confirm connectivity to HAProxy and brokers

telnet gateway.kafka.net 6677
telnet kafka-1.kafka.net 6668

Final Checklist

Ensure all brokers have consistent Kerberos configurations.
Verify the client-side security.protocol and sasl.kerberos.service.name settings.
Ensure HAProxy uses TCP passthrough (mode tcp) for Kerberos.

With these adjustments, the Kerberos authentication error should be resolved. Let me know if further clarification is needed!

Happy hadooping

Re: How to set Kerberos Kafka + Haproxy (Load Balancer)

arraiano78 — Thu, 23 Jan 2025 17:16:41 GMT

Hello

@polingsky202 ,

i'm facing the same problem and had the same errors on logs implementing HAproxy with 3 brokers.

Have you solved this issue ?

Thank you for your help.

Best regards.