​Troubleshooting kerberos issues in kafka for producer/consumers

[#1]

In kafka there are three types of communication :

1. Between brokers

2. Between client and broker.

3. Broker to zookeeper

In order to communicate in kerberos enabled cluster one needs to authenticate itself. So when broker will try to communicate with other broker in the cluster it will need to authenticate first. Same for the clients communicating to the brokers.

In Kafka, JAAS files is used for authentication. Lets first understand what's there in JAAS file. In kafka there are two JAAS files :

1. kafka_jaas.conf

2. kafka_client_jaas.conf.

Let's discuss about kafka_jaas.conf. This file will be used for authentication when a broker in a cluster tries to communicate with other brokers in cluster. Take a look at the content and understand its purpose :

KafkaServer { 
com.sun.security.auth.module.Krb5LoginModule required        
useKeyTab=true        
keyTab="/etc/security/keytabs/kafka.service.keytab"        
storeKey=true        
useTicketCache=false        
serviceName="kafka"        
principal="kafka/c6401.ambari.apache.org@EXAMPLE.COM";
}; 

KafkaServer section will be used by broker for authentication when it tries to communicate with other brokers in cluster. It should always be configured to use keytab and principal.

Value of `serviceName` should be the principal as which kafka is running.

`storeKey=true` : significance of setting the storekey parameter to true in jaas.conf

Client { // used for zookeeper connection
com.sun.security.auth.module.Krb5LoginModule required       
useKeyTab=true        
keyTab="/etc/security/keytabs/kafka.service.keytab"        
storeKey=true        
useTicketCache=false        
serviceName="zookeeper"        
principal="kafka/c6401.ambari.apache.org@EXAMPLE.COM";
}; 

Client section in kafka_jaas.conf will be used for authentication when broker wants to communicate with zookeeper. It should always be configured to use keytab and principal.

Value of `serviceName` should be the principal as which zookeeper service is running.

kafka_client_jaas.conf will be used by clients(producer/consumer) to authenticate to kafka broker. kafka_client_jaas.conf has two sections, take a look :

KafkaClient {      
com.sun.security.auth.module.Krb5LoginModule required      
useTicketCache=true      
renewTicket=true      
serviceName="kafka";     
}; 

Client {
com.sun.security.auth.module.Krb5LoginModule required      
useTicketCache=true      
renewTicket=true      
serviceName="zookeeper";
}; 

KafkaClient section : As the name suggest it will be used when client wants to communicate to broker.

Value of `serviceName` should be the principal as which kafka is running. You can configure it to use ticket cache or keytab and principal.

Client : This part of JAAS file is only used by clients which are using old consumer api. In old consumer api, consumers need to connect to zookeeper.

In new consumer api clients communicate to brokers instead of zookeeper. Hence while authentication it will use KafkaClient section in kafka_client_jaas.conf.

Producers will always use KafkaClient section in kafka_client_jaas.conf as it will send request to broker node. For long running kafka clients it recommended to configure JAAS file to use keytab and principal.

Please refer below example :

KafkaClient {      
com.sun.security.auth.module.Krb5LoginModule required      
useKeyTab=true      
keyTab="/etc/security/keytabs/storm.service.keytab"      
storeKey=true
useTicketCache=false      
serviceName="kafka"      
principal="storm@EXAMPLE.COM";
};

When kerberos is integrated with kafka we see lot of issues while trying to produce/consume messages to kafka.

There are instances where client throws a generic error and we don't know what's going wrong. To tackle such conditions I will discuss about the checks which need to be done when you face such issue :

1. make sure kerberos client is installed on the node.

2. Check if you can obtain a ticket of the principal : -

If you want to obtain TGT using password :

# kinit <principalName> 

If you are using keytab check if user has permission to read the keytab : -

Using keytab :

# kinit -kt /Path/to/Keytab <PrincipalName> 

3. Confirm if you a ticket in ticket cache which is not expired :

# klist 

4. Confirm if user has read permission on JAAS file which is used.

5. Confirm if client can communicate to kafka broker and port on which broker is listening :

# ping <broker.hostname>
# telnet  broker.hostname:port 

6. Check if you are using correct security protocol `--security-protocol` as configured in server.properties in broker.

7. Try exporting JAAS file and run producer/consumer again :

# export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/Path/to/Jaas/File" 

In order to enable debug for kerberos :

# export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/Path/to/Jaas/File -Dsun.security.krb5.debug=true" 

8. Still if you are facing authentication issue, try enabling debug for console-producer/console-consumer on kafka client node : As root user :

# vim /usr/hdp/current/kafka-broker/config/tools-log4j.properties log4j.rootLogger=DEBUG, stderr 

In debug logs you should see which principal, security protocol is used and to which broker request is being send.

9. Once you are confirmed that authentication is working fine it time for you to confirm if user has the required permission on the topic. If kafka is configured to use kafka acl's,

please refer below link : Authorization commands

If kafka is configured to use ranger, make sure policy is defined for the topic and principal