Reply
New Contributor
Posts: 5
Registered: ‎08-21-2018

Re: Console producer/consumer not working in kafka 0.10.2 with kerberos

[ Edited ]

Hi,

 

Follow below steps.

1)change Inter Broker Protocol property to SASL_PLAINTEXT in Cloudera manager kafka configuration. and restart kafka service.

 

2)create jaas.conf file in your home path /home/userid/
vi jaas.conf
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=false
useKeyTab=true
serviceName="kafka"
StoreKey=true

#create new keytab by using principal of same user id and put it in below path
keyTab="/home/userid/useridkerberos.keytab"

#below replace with your correct principal name
principal="userid@REALHOSTNAME.COM"
client=true;};

 

3)Create client.properties file containing the following properties in the same path /home/userid/
sudo vi client.properties
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka

 

4) Login with same user id's login and do kinit

kinit -kt useridkerberos.keytab userid@REALMHOSTNAME.COM

 

5) Cretaing Topics:
----------------
/usr/bin/kafka-topics --create --zookeeper hostname1:2181,hostname2:2181,hostname3:2181/kafka --replication-factor 2 --partitions 2 --topic newtopic1

 

6) Describing Topics:
------------------

/usr/bin/kafka-topics --describe --zookeeper hostname1:2181,hostname2:2181,hostname3:2181/kafka --topic testtopic1

 

7)  export KAFKA_OPTS="-Djava.security.auth.login.config=/home/userid/jaas.conf"
verify it:  echo "$KAFKA_OPTS"

 

8) Writting message using Producer:
--------------------------------
/usr/bin/kafka-console-producer --broker-list brokerhostname1:9092,brokerhostname2:9092 --topic newtopic6 --producer.config client.properties

 

9) Open duplicate session of smae machine and run consumer command

Reading message using Consumer:
-------------------------------

export KAFKA_OPTS="-Djava.security.auth.login.config=/home/userid/jaas.conf"


/usr/bin/kafka-console-consumer --new-consumer --topic newtopic6 --from-beginning --bootstrap-server brokerhostname1:9092,brokerhostname2:9092 --consumer.config client.properties

 

 

try it. It may work at your end also!!

 

 

 

New Contributor
Posts: 4
Registered: ‎08-29-2018

Re: Console producer/consumer not working in kafka 0.10.2 with kerberos

poojary_sudth:

Thank you, but that is +- the same set of tasks I did before.

 

I found how to make consumer work, it was necessary to add  parameter  --partition 0:

KAFKA_OPTS="-Djava.security.auth.login.config=/root/jaas.conf" kafka-console-consumer --bootstrap-server ourHost:9092 --topic test --consumer.config /root/client.properties --partition 0

 I cannot see all the messages comming into the topic, but at least some of them which fall into specified partition are printed. Which is enought for me to confirm that Kafka broker works. 

I found this hint here:  https://stackoverflow.com/questions/34844209/consumer-not-receiving-messages-kafka-console-new-consu...

Announcements
New solutions