Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to run kafka with kerberos?

How to run kafka with kerberos?

New Contributor

I am new to Kafka.

What I am trying to do is to run Kafka with Kerberos.

I have some questions about this.

Should I run Zookeeper and Kafka with different os users?

While trying to run Kafka with Kerberos, I had done some changes in config files following documentations. Do I have to do anything in zookeeper config files?

9 REPLIES 9

Re: How to run kafka with kerberos?

@Utku Utku Refer below documentation to configure Kafka for Kerberos.By default, Kafka and Zookeeper will run as different users i.e. kafka and zookeeper respectively.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_secure-kafka-ambari/content/ch_secure-ka...

Re: How to run kafka with kerberos?

New Contributor

In kafka environment, I had changed some parameters in server.properties file for enabling SASL and then created the jaas file for kafka.

In zookeeper side, I also did some changes so that zookeeper runs with a jaas file.

now I am trying to solve some issues about kerberos.

when there is some progress, I will write them here.

Re: How to run kafka with kerberos?

Contributor

Here are steps for a working Kafka on Kerberized HDP cluster:

1.Add kafka services from Ambari.

2.Select Kafka-broker hosts :

3.Customized Kafka broker properties and Add following additional properties:

allow.everyone.if.no.acl.found=true

principal.to.local.class=kafka.security.auth.KerberosPrincipalToLocal

sasl.enabled.mechanisms=GSSAPI

sasl.mechanism.inter.broker.protocol=GSSAPI

security.inter.broker.protocol=SASL_PLAINTEXT

security.protocol=SASL_PLAINTEXT

super.users=user:kafka,user1

5.Change Listener as shown below:

SASL_PLAINTEXT://localhost:6667

6.Complete the installation.

7.Verify Installation:

Make sure that following properties are added into producer.properties and consumer.properties file: (if not add manually)

sasl.mechanism=GSSAPI

security.protocol=SASL_PLAINTEXT

Get Kerberos TGT:

If testing as kafka user:

#kinit kafka/hdp-qa-n1.example.com@EXAMPLE.COM -kt /etc/security/keytabs/kafka.service.keytab

As user1 user:

[user1@hdp-qa-n1 conf]$ kinit

Password for user1@EXAMPLE.COM:

Create a topic:

$/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper hdp-qa-n1.example.com:2181 --create --topic testtopic --partitions 2 --replication-factor 1

Start producer console:

$/usr/hdp/current/kafka-broker/bin /kafka-console-producer.sh --broker-list hdp-qa-n1.example.com:6667 --topic testtopic --security-protocol SASL_PLAINTEXT

Start consumer console:

$/usr/hdp/current/kafka-broker/bin /kafka-console-consumer.sh --zookeeper hdp-qa-n1.example.com:2181 --topic testtopic22 --from-beginning --security-protocol SASL_PLAINTEXT

You should be able to get message from producer to consumer console as shown below:

[kafka@hdp-qa-n1 bin]$ ./kafka-console-producer.sh --broker-list hdp-qa-n1.example.com:6667 --topic testtopic22 --security-protocol SASL_PLAINTEXT

hi

hello

[kafka@hdp-qa-n1 bin]$ ./kafka-console-consumer.sh --zookeeper hdp-qa-n1.example.com:2181 --topic testtopic22 --from-beginning --security-protocol SASL_PLAINTEXT

{metadata.broker.list=hdp-qa-n1.example.com:6667,hdp-qa-edge.example.com:6667, request.timeout.ms=30000, client.id=console-consumer-8642, security.protocol=SASL_PLAINTEXT}

hi

hello

[kafka@hdp-qa-edge bin]$ ./kafka-console-consumer.sh --zookeeper hdp-qa-n1.example.com:2181 --topic testtopic22 --from-beginning --security-protocol SASL_PLAINTEXT

{metadata.broker.list=hdp-qa-n1.example.com:6667,hdp-qa-edge.example.com:6667, request.timeout.ms=30000, client.id=console-consumer-50688, security.protocol=SASL_PLAINTEXT}

hi

hello

user1@hdp-qa-edge bin]$ ./kafka-console-consumer.sh --zookeeper hdp-qa-n1.example.com:2181 --topic testtopic22 --from-beginning --security-protocol SASL_PLAINTEXT

{metadata.broker.list=hdp-qa-n1.example.com:6667,hdp-qa-edge.example.com:6667, request.timeout.ms=30000, client.id=console-consumer-76108, security.protocol=SASL_PLAINTEXT}

hi

hello

I have not covered ACL part here. Allowing everyone to produce and consume message.

Re: How to run kafka with kerberos?

Expert Contributor

@khireswar Kalita,

Thanks, this was very informative .. however, a basic question (since i'm new to kafka & kerberos)

When i tried the same with user - kafka1, i'm unable to get the kerberos token

-----------

[kafka1@sandbox ~]$ kinit kinit: Client not found in Kerberos database while getting initial credentials

---------------

How do i add kafka1 to kerberos database ?

(i.e. how did you add user - user1 to kerberos database ?)

Re: How to run kafka with kerberos?

Expert Contributor

@khireswar Kalita, i was able to add the user to the Kerberos database

kadmin.local -q "addprinc kafka1"

Re: How to run kafka with kerberos?

Expert Contributor

@khireswar Kalita,i followed the steps above, but now i'm getting the following error on the Kafka console producer

---------------------------

[kafka1@sandbox ~]$ $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic kafka1_topic2 --security-protocol SASL_PLAINTEXT hi [2016-11-28 00:12:29,361] WARN Error while fetching metadata [{TopicMetadata for topic kafka1_topic2 -> No partition metadata for topic kafka1_topic2 due to kafka.common.TopicAuthorizationException}] for topic [kafka1_topic2]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo)

------------------------

Any ideas on this ?

Highlighted

Re: How to run kafka with kerberos?

Contributor

Looks like this seems to be access control issue:

1. Can you check ACL properties allow.everyone.if.no.acl.found=true

2. Did you enable ranger plugin for kafka?

Re: How to run kafka with kerberos?

Expert Contributor

@khireswar Kalita,

I seem to be getting separate error when i try to publish to kafka topic, when logged in as root

Error - kafka.common.LeaderNotAvailableException

Details in the link :

https://community.hortonworks.com/questions/68641/kerberized-hdp-24-kafka-giving-error-kafkacommonle...

I'd fixed this earlier by adding property --> advertised.host.name=sandbox.hortonworks.com,

but now the probem has re-occurred (though the property is still there)

Any ideas on this ?

Re: How to run kafka with kerberos?

Contributor