Kerberized HDP 2.4 - can i use users kafka1 for publish/subscribe to Kafka topic

[#1]

Hi Kafka, HDP experts,

I've a Kerberized HDP 2.4 sandbox..

i'm able to do kinit for the principal user (kafka), and publish & subscribe to topic.

How do i do the same with additional users (e.g. kafka1) ?

I've a Unix user - kafka1 setup.. How do i obtain a kinit for kafka1 user, so i can publish/subscribe messages to Kafka topic ?

[#2]

@Ayub Khan, @emaxwell - any inputs on this ?

[#3]

@Karan Alang Seems like 'kafka1' is a unix user and your mit kdc does not know about it. So you have register user with kdc and create keytab for the user. Please follow the below steps.

1. Run kadmin.local

2. In the kadmin prompt, run this command 'addprinc -randkey kafka1'

3. Then this command: 'xst -k /tmp/keytabs/kafka1.headless.keytab kafka1'

This should actually create the keytab and register user kafka1 in kdc.

Now you can do the kinit with the kafka1 keytab(from step2) and perform the operations.

If you have ranger configured in your cluster for kafka, you need to add appropriate policies in ranger.

Hope this helps.

[#4]

@Karan Alang Is this working for you? Are you able to publish to topic?

[#5]

@Ayub Khan,

thanks, i was able to add the user kafka1, kafka2 - to KDC and get Kerberos token for them.

However, now i'm facing issue wherein i'm not able to publish to Kafka topic, even when i'm logged in as root (where i believe it uses service user - kafka)

When i publish to the topic, i'm getting this error ->

-------------------------------------------------------------

[root@sandbox libs]# $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic kafka1_topic2 --security-protocol SASL_PLAINTEXT hello [2016-11-28 06:44:03,499] WARN Error while fetching metadata [{TopicMetadata for topic kafka1_topic2 -> No partition metadata for topic kafka1_topic2 due to kafka.common.LeaderNotAvailableException}] for topic [kafka1_topic2]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)

---------------------------------------------------------------

Details are in this link ->

https://community.hortonworks.com/questions/68641/kerberized-hdp-24-kafka-giving-error-kafkacommonle...

Couple of days back, i'd got this issue & added the following constant to fix the issue -

advertised.host.name=sandbox.hortonworks.com

host.name=sandbox.hortonworks.com

But the issue has re-occurred (above fix still in place), and not sure how to fix this.

Any ideas on this ?

[#6]

@Karan Alang Could you please try kiniting and running the following command:

export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf" 

After that run the producer. Also I believe you have enabled Ranger-Kafka plugin. Have you enabled SSL too?

[#7 · Accepted solution]

@dbains, @Ayub Khan- i seem to have fixed this issue for now ..

I've created another cluster, and repeated the steps for kerberization .. and am able to publish/consume messages to kafka topics. I'll update if i'm able to fix the above error in the original cluster.

But first, i need to be able to create/limit access to users kafka1, kafka2 to specific kafka topics.

[#8]

@Karan Alang

In the original cluster are you using Ranger for authorization or ACLs?

[#9]

@dbains, in the original cluster, i'm using Ranger for authorization.