Configuring two listeners in Kerberized Kafka

[#1]

I have configured two listeners for non-kerberized Kafka before and it works out fine. That configuration for every broker is as follows:

security.inter.broker.protocol=PLAINTEXT
listeners=PLAINTEXT://<internal-ip>:9092,EXTERNAL://<external-ip>:19092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

I checked the actual configuration file kafka.properties used by running Kafka process and it is as follows:

listeners=PLAINTEXT://<broker-hostname>:9092,
...
...
listeners=PLAINTEXT://<internal-ip>:9092,EXTERNAL://<external-ip>:19092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

Though it includes two same entries for listeners, while I am not sure whether this is a bug or not, our Kafka works as expected.  The problem is I use the similar configuration for new Kerberized Kafka but it cannot work. The configuration is as follows:

security.inter.broker.protocol=SASL_PLAINTEXT
listeners=SASL_PLAINTEXT://<broker-hostname>:9092,
...
...
listeners=SASL_PLAINTEXT://<internal-ip>:9092,EXTERNAL://<external-ip>:19092
listener.security.protocol.map=SASL_PLAINTEXT:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT

I believe the configuration does not work based on that when I comment all broker hostnames in /etc/hosts kafka-console-consumer/producer scripts raise error:

9/08/28 10:32:01 ERROR clients.NetworkClient: [Producer clientId=console-producer] Connection to node -1 failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
19/08/28 10:32:01 ERROR internals.ErrorLoggingCallback: Error when sending message to topic test1 with key: null, value: 3 bytes with error:
org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:361)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:359)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:359)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:269)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:206)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:81)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:474)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:481)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
	at java.lang.Thread.run(Thread.java:748)
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
	... 14 more
Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
	at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
	at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
	... 17 more

Does anybody else run into this ? I could really use some help. Thanks.

[#2]

Hi @iamabug 

Before running the consumer, could you please add to your terminal:

export KAFKA_OPTS="-Dsun.security.krb5.debug=true"

After that, share the DEBUG info for further review.

Thanks.

[#3]

Sorry @ManuelCalvo , I probably have no time to do that because the test environment has been torn down. And I kind of have a theory for that. I guess that Kerberos principals are based on hostnames while I have changed Kafka listeners to IP addresses. With that config, Kafka clients cannot build a qualified principal name because they do not know hostnames of brokers. Thanks for you help anyway !