ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

[#1]

Hello,

I have enable kerberos on cluster. I can successfully connect to consumer through kafka-konsole-consumer.

But when i try to connect to topic through NiFi ConsumeKafka processor i have error:

WARN [Timer-Driven Process Thread-8] o.a.n.p.kafka.pubsub.ConsumeKafka_1_0 ConsumeKafka_1_0[id=504e5811-0168-1000-0000-000024c83cc5] Was interrupted while trying to communicate with Kafka with lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@51911c24. Will roll back session and discard any partially received data.   WARN [kafka-kerberos-refresh-thread-nifi/*@*] o.a.k.c.security.kerberos.KerberosLogin [Principal=nifi/*@*]: TGT renewal thread has been interrupted and will exit.

[#2]

@Geoffrey Shelton Okot

Can i remove zookeeper's trees? Will they be recreate after deleting with right permission?

[#3]

@ Ruslan Fialkovsky

Can you compare these files on both cluster, if it doesn't exist on the DEV then create them with correct owner and privileges

Kafka

/etc/kafka/conf/kafka_client_jaas.conf 
/etc/kafka/conf/kafka_jaas.conf

Zookeeper

/etc/zookeeper/conf/zookeeper_jaas.conf
/etc/zookeeper/conf/zookeeper_client_jaas.conf

After updating these files restart the zk and Kafka and retry

HTH

[#4]

@Geoffrey Shelton Okot

I compared files, yes their exists and the same.

[#5]

@ Ruslan Fialkovsky

Logon to Zk on DEV and PROD Check whether you have child znodes under

 ls /config/topics 

Validate that you have Kerberos entries in /etc/nifi/conf/nifi_jaas.conf

example

NiFiClient { 
    com.sun.security.auth.module.Krb5LoginModule required 
    useKeyTab=true keyTab="/etc/security/keytabs/nifi.service.keytab" 
    storeKey=true 
    useTicketCache=false 
    principal="nifi/{kdc_host}@REALM"; 
}; 
RegistryClient { 
    com.sun.security.auth.module.Krb5LoginModule required 
    useKeyTab=true keyTab="/etc/security/keytabs/nifi.service.keytab" 
    storeKey=true 
    useTicketCache=false 
    principal="nifi/{kdc_host}@REALM";

Please revert

[#6]

@Geoffrey Shelton Okot

Yes i have

ls /config/topics
[test1]

NiFiClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytabs/nifi.service.keytab"
  storeKey=true
  useTicketCache=false
  principal="nifi/[email protected]";
};
RegistryClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytabs/nifi.service.keytab"
  storeKey=true
  useTicketCache=false
  principal="nifi/[email protected]";

[#7]

@Ruslan Fialkovsky

I think the topics created before changing the ACL's don't inherit the permissions. From the znode entry, I see you have one topic called test.

To validate can you create a new topic and ensure you change them in your Nifi flow files and retest! Normally if you run rmr in zookeeper CLI the entry deleted should be created but I am not sure for Kafka topics.

Please revert

[#8]

@Geoffrey Shelton Okot

Thank you, you are right, the problem was really in zookeeper's acl.

I copied everything in "ZooKeeper directory" from Test cluster to Dev cluster and that was help. But i don't know what exactly permission affected it. Is something way to get list all acl permission by Zookeeper? I would like to compare it with all acl from both cluster.