Connecting to on-prem kafka cluster from cloud AWS using Kerberos auth

[#1]

Is it possible to connect to on-prem kafka cluster using Kerberos authentication from cloud deployed service?

When we are trying to connect we are getting below error:

Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm "ABC.COM"

This is my jaas config:

com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/pathtokeytab" principal="principal_name@ABC.COM" ;

Please help me if anyone has faced such error. 

Full stack trace:

Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Cannot locate KDC
        at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:162)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99)
        at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:441)
        at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:422)
        ... 3 more
Caused by: javax.security.auth.login.LoginException: Cannot locate KDC
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)
        at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:103)
        at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:61)
        at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:104)
        at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:151)
        ... 8 more
Caused by: KrbException: Cannot locate KDC
        at sun.security.krb5.Config.getKDCList(Config.java:1084)
        at sun.security.krb5.KdcComm.send(KdcComm.java:218)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 25 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm "ABC.COM"

[#2]

Hi, @Ankit88 ,

Is your cloud Kafka running on CDP Public Cloud or is it your own deployment on AWS?

What about the Kafka on-prem? Is it a Kafka on CDP or some other type of deployment?

What the version of the on-prem Kafka?

André

[#3]

Kafka version on prem -2.8.0

both cloud and on prem kafka running on regular unix box .Installed manually.

On cloud , kafka is installed on Ec2 box.

[#4 · Accepted solution]

Hi, @Ankit88 , thanks for the info!

The error that you're getting is because the host in the cloud where your application is running doesn't know anything about your Kerberos configuration, in particular, where your KDC is.

There's a few things you must do to configure it properly:

• Ensure the Kerberos client libraries are installed on that host (krb5-workstation package)
• Your on-prem krb5.conf file must be copied to the cloud host. If you also have a Kerberos KDC in the cloud, there will already be a krb5.conf file on that host and the two configurations will need to be carefully merged.
  • The sections [realms] and [domain_realm] are especially important to solve your issue.
• Ensure that the hostname of your KDC as well as the host names of ALL Kafka brokers can be resolved from the cloud (you can test it with nslookup and/or ping). This must work correctly for Kerberos to work. If there's no integrated DNS you will have to add entries to your /etc/hosts file to ensure the resolution is correct.
• Ensure that any firewalls are configured correctly to open ports between your application and your on-prem environment:
  • Open all the ports required for the client to connect to Kafka
  • Open all the ports required for the client to communicate with the KDC (typically, ports 88 UDP and 88 TCP)

The above being correctly configuration, you should be able to authenticate correctly using Kerberos. A simple test to ensure it's working, before you try the Kafka application, is to authenticate on the command line using the kinit command.

Hope this helps.

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#5]

Hi @araujo

I got the problem solved as entry was missing in etc/host file, and able to send msg to kafka topic using sh script . But , when I try to send the message using java code I get below error :

Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: KDC has no support for encryption type (14))
        at java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:778)
        at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:266)
        at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:196)
        at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:209)
        ... 14 common frames omitted
Caused by: sun.security.krb5.KrbException: KDC has no support for encryption type (14)
        at java.security.jgss/sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:69)
        at java.security.jgss/sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:224)
        at java.security.jgss/sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:235)
        at java.security.jgss/sun.security.krb5.internal.CredentialsUtil.serviceCredsSingle(CredentialsUtil.java:482)
        at java.security.jgss/sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:340)
        at java.security.jgss/sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:314)
        at java.security.jgss/sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:169)
        at java.security.jgss/sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:493)
        at java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:700)
        ... 17 common frames omitted
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected value (906)
        at java.security.jgss/sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
        at java.security.jgss/sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
        at java.security.jgss/sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
        at java.security.jgss/sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:54).

Please help me in solving the error.

I tried changing default_tkt_enctypes = rc4-hmac but it did not work

[#6]

@Ankit88 ,

For a Java application you need to add the following Kafka property to your producer/consumer configuration:

sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="./myuser.keytab" principal="myuser@XYZ.SITE";

 Where myuser.keytab is your user's Kerberos keytab.

Cheers,

André