Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Configuring two listeners in Kerberized Kafka

Configuring two listeners in Kerberized Kafka

Explorer

I have configured two listeners for non-kerberized Kafka before and it works out fine. That configuration for every broker is as follows:

 

 

security.inter.broker.protocol=PLAINTEXT
listeners=PLAINTEXT://<internal-ip>:9092,EXTERNAL://<external-ip>:19092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

 

 

I checked the actual configuration file kafka.properties used by running Kafka process and it is as follows:

 

 

listeners=PLAINTEXT://<broker-hostname>:9092,
...
...
listeners=PLAINTEXT://<internal-ip>:9092,EXTERNAL://<external-ip>:19092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

 

 

Though it includes two same entries for listeners, while I am not sure whether this is a bug or not, our Kafka works as expected.  The problem is I use the similar configuration for new Kerberized Kafka but it cannot work. The configuration is as follows:

 

 

security.inter.broker.protocol=SASL_PLAINTEXT
listeners=SASL_PLAINTEXT://<broker-hostname>:9092,
...
...
listeners=SASL_PLAINTEXT://<internal-ip>:9092,EXTERNAL://<external-ip>:19092
listener.security.protocol.map=SASL_PLAINTEXT:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT

 

 

I believe the configuration does not work based on that when I comment all broker hostnames in /etc/hosts kafka-console-consumer/producer scripts raise error:

 

 

9/08/28 10:32:01 ERROR clients.NetworkClient: [Producer clientId=console-producer] Connection to node -1 failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
19/08/28 10:32:01 ERROR internals.ErrorLoggingCallback: Error when sending message to topic test1 with key: null, value: 3 bytes with error:
org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:361)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:359)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:359)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:269)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:206)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:81)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:474)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:481)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
	at java.lang.Thread.run(Thread.java:748)
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
	... 14 more
Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
	at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
	at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
	... 17 more

 

 

Does anybody else run into this ? I could really use some help. Thanks.

2 REPLIES 2
Highlighted

Re: Configuring two listeners in Kerberized Kafka

Contributor

Hi @iamabug 

 

Before running the consumer, could you please add to your terminal:

 

export KAFKA_OPTS="-Dsun.security.krb5.debug=true"

 

After that, share the DEBUG info for further review.

 

Thanks.

Re: Configuring two listeners in Kerberized Kafka

Explorer

Sorry @ManuelCalvo , I probably have no time to do that because the test environment has been torn down. And I kind of have a theory for that. I guess that Kerberos principals are based on hostnames while I have changed Kafka listeners to IP addresses. With that config, Kafka clients cannot build a qualified principal name because they do not know hostnames of brokers. Thanks for you help anyway !