Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka producer / Java debugging

Kafka producer / Java debugging

New Contributor

Hi - this is probably more a kerberos question than a Kafka issue - but will ask anyway.

I have a Kafka producer which intermittently fails with "Client not found in Kerberos database"

It works most of the time so suspect issue is with one of the KDCs (Active Directory), so I want to use debug logging to capture which KDC is giving the issue. The enterprise uses a DNS label for KDCs (ie KERBEROS.GROUP.COM) which returns a random list of 20+ KDCs, so each connection attempt uses a different IP.

If I set KRB5_TRACE variable on Linux, and use Linux kinit - I see the below type of output:

[26116] 1490023255.280436: Sending request (305 bytes) to KERBEROS.GROUP.COM

[26116] 1490023255.280462: Resolving hostname KERBEROS.GROUP.COM

[26116] 1490023255.280567: Sending initial UDP request to dgram 10.x.x.x:88

[26116] 1490023255.282137: Received answer from dgram 10.x.x.x:88

which contains the IP address and is exactly what I want.

However, I can't figure out how to get the IP info from a Java based Kafka Producer. On the JVM I can set "-Dsun.security.krb5.debug=true", but the logging then looks like:

>>> KrbKdcReq send: kdc=KERBEROS.GROUP.COM UDP:88, timeout=30000, number of retries =3, #bytes=1438

>>> KDCCommunication: kdc=KERBEROS.GROUP.COM UDP:88, timeout=30000,Attempt =1, #bytes=1438

>>> KrbKdcReq send: #bytes read=131

>>> KrbKdcReq send: kdc=KERBEROS.GROUP.COM TCP:88, timeout=30000, number of retries =3, #bytes=1438

>>> KDCCommunication: kdc=KERBEROS.GROUP.COM TCP:88, timeout=30000,Attempt =1, #bytes=1438

>>>DEBUG: TCPClient reading 1600 bytes

i.e. no IP address recorded. I'm tying to establish which specific KDC the request has gone to. Any ideas how to capture the IP in the Java debuging options?

1 REPLY 1

Re: Kafka producer / Java debugging

Cloudera Employee
@cmcbugg

You can use the following jvm options to add additional logging for Kerberos:

-Dsun.security.krb5.debug=true -Dsun.security.jgss.debug=true -Djavax.net.debug=all

When you run your kafka process at DEBUG log level, it should ideally log the kerberos ticket exchange, which will contain the KDC coordinates.

Hope this helps