Hello:
How to use HAProxy to connect for Kafka with Kerberos authentication?
I have three kafka brokers, and i try to use haproxy in front of kafka, but kerberos authenticated failed
My haproxy.conf
listen kafka
bind *:6677
mode tcp
balance roundrobin
server kafka1 kafka-1.kafka.net:6668 check
server kafka2 kafka-2.kafka.net:6669 check
server kafka3 kafka-3.kafka.net:6666 check
I also modified
kafka1 server.properties
- advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6668
- listeners=INTERNAL://:6667,LB://:6668
- listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
- inter.broker.listener.name=INTERNAL
- listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET"
kafka2 server.properties
- advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6669
- listeners=INTERNAL://:6667,LB://:6669
- listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
- inter.broker.listener.name=INTERNAL
- listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET";
kafka3 server.properties
- advertised.listeners=INTERNAL://:6667,LB://gateway.kafka.net:6666
- listeners=INTERNAL://:6667,LB://:6666
- listener.security.protocol.map=INTERNAL:SASL_PLAINTEXT,LB:SASL_PLAINTEXT
- inter.broker.listener.name=INTERNAL
- listener.name.LB.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.service.keytab"principal="kafka/gateway.kafka.net@KAFKA.NET";
amd use the command
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --topic my-topic --broker-list gateway.kafka.net:6677 --producer-property security.protocol=SASL_PLAINTEXT
Will get the error:
[2024-10-08 20:07:58,330] ERROR [Producer clientId=console-producer] Connection to node -1 failed authentication due to: Authentication failed due to invalid credentials with SASL mechanism GSSAPI (org.apache.kafka.clients.NetworkClient)
[2024-10-08 20:07:58,330] ERROR Error when sending message to topic my-topic5 with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
