Skip to main content
ururu
ururu
Gold ⭐️
January 17, 2019
Solved

ConsumeKafka processor's NiFi can't communicate with Kafka in kerberized cluster

Hello,

I have enable kerberos on cluster. I can successfully connect to consumer through kafka-konsole-consumer.

But when i try to connect to topic through NiFi ConsumeKafka processor i have error:

WARN [Timer-Driven Process Thread-8] o.a.n.p.kafka.pubsub.ConsumeKafka_1_0 ConsumeKafka_1_0[id=504e5811-0168-1000-0000-000024c83cc5] Was interrupted while trying to communicate with Kafka with lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@51911c24. Will roll back session and discard any partially received data. WARN [kafka-kerberos-refresh-thread-nifi/*@*] o.a.k.c.security.kerberos.KerberosLogin [Principal=nifi/*@*]: TGT renewal thread has been interrupted and will exit.

Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
January 23, 2019

@ Ruslan Fialkovsky

Can you log on both cluster as zookeeper run kinit then

$ /usr/hdp/current/zookeeper-server/bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] getAcl /config/topics

Compare the outputs there should be a difference between the 2 cluster

sample output

[zk: localhost:2181(CONNECTED) 1] getAcl /config/topics
'world,'anyone
: r
'sasl,'kafka
: cdrwa

And on the other

[zk: localhost:2181(CONNECTED) 1] getAcl /config/topics
'world,'anyone
: cdrwa

If that's the case have a look at how to set the correct ACL for the /config/topics in this HCC document

Please let me know

View in thread

16 replies

ururu
ururuAuthor
Gold ⭐️
January 22, 2019

I can't connect with --bootstrap-server key, only with --zookeeper works. Obviously "ConsumeKafka processor" use "bootstrap" mod.

Can i use zookeeper with "ConsumeKafka processor" or how can i debug why i can't to connect direct to broker?

kafka-console-consumer.sh --bootstrap-server server:6667 --topic test5 --from-beginning --security-protocol SASL_PLAINTEXT

don't work

kafka-console-consumer.sh --zookeeper server:2181--topic test5 --from-beginning --security-protocol SASL_PLAINTEXT

work fine

Translate
Shelton
Shelton
Gold ⭐️⭐️⭐️
January 22, 2019

@ Ruslan Fialkovsky

Is your nifi part of the kerberized cluster or is a separate HDF cluster? What are versions of the components NIF,HDP etc ?

Translate
ururu
ururuAuthor
Gold ⭐️
January 22, 2019

@Geoffrey Shelton Okot

I use HDF on HDP as one cluster, yes is kerberized. Ambari-2.6.2.2, HDP-2.6.5, HDF-3.1.2.

Translate
Shelton
Shelton
Gold ⭐️⭐️⭐️
January 23, 2019

@ Ruslan Fialkovsky

Are you doing some lab, this looks familiar can you share the link, I want to see if I can reproduce it on my cluster?

Translate
ururu
ururuAuthor
Gold ⭐️
January 23, 2019

@Geoffrey Shelton Okot

It's real case at my work. I configured Test cluster from Hortonworks documentation and everything works good but at the same time Development cluster with same configuration doesn't work and i don't understand why it is. I don't have lab or some special instruction but i can to show you anything my config files or screenshots. Is it maybe some problem with zookeeper?

Translate
Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
January 23, 2019

@ Ruslan Fialkovsky

Can you log on both cluster as zookeeper run kinit then

$ /usr/hdp/current/zookeeper-server/bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] getAcl /config/topics

Compare the outputs there should be a difference between the 2 cluster

sample output

[zk: localhost:2181(CONNECTED) 1] getAcl /config/topics
'world,'anyone
: r
'sasl,'kafka
: cdrwa

And on the other

[zk: localhost:2181(CONNECTED) 1] getAcl /config/topics
'world,'anyone
: cdrwa

If that's the case have a look at how to set the correct ACL for the /config/topics in this HCC document

Please let me know

Translate
    ururu
    ururuAuthor
    Gold ⭐️
    January 23, 2019

    @Geoffrey Shelton Okot

    Working Test:

    [zk: localhost:2181(CONNECTED) 0] getAcl /config/topics

    'world,'anyone

    : r

    'sasl,'kafka

    : cdrwa

    [zk: localhost:2181(CONNECTED) 1]

    Not working Dev:

    [zk: localhost:2181(CONNECTED) 0] getAcl /config/topics

    'world,'anyone

    : cdrwa

    [zk: localhost:2181(CONNECTED) 1]

    Translate
    Shelton
    Shelton
    Gold ⭐️⭐️⭐️
    January 23, 2019

    @ Ruslan Fialkovsky

    Now you see that on DEV world: everyone has cdrwa while for kafka there is no entry so you will need to setAcl for kafka

    setAcl /config/topics world:anyone:cdrwa,sasl:kafka:cdrwa

    Did you grasp how to use the using superDigest to become a Zookeeper superuser, that's like the zookeeper root user who can add/change the ACL's ,let me know if you need help on that.

    Translate
    ururu
    ururuAuthor
    Gold ⭐️
    January 23, 2019

    @Geoffrey Shelton Okot

    I set next acl on Dev cluster:

    [zk: localhost:2181(CONNECTED) 1] getAcl /brokers
    'world,'anyone
    : cdrwa
    'sasl,'kafka
    : cdrwa
    [zk: localhost:2181(CONNECTED) 2] getAcl /controller
    'world,'anyone
    : r
    'sasl,'kafka
    : cdrwa
    [zk: localhost:2181(CONNECTED) 3] getAcl /config
    'world,'anyone
    : cdrwa
    'sasl,'kafka
    : cdrwa
    [zk: localhost:2181(CONNECTED) 4] getAcl /config/topics
    'world,'anyone
    : cdrwa
    'sasl,'kafka
    : cdrwa

    but kafka-console-consumer.sh --bootstrap-server still not work.

    Translate
    Shelton
    Shelton
    Gold ⭐️⭐️⭐️
    January 23, 2019

    @ Ruslan Fialkovsky

    Can you compare these files on both cluster, if it doesn't exist on the DEV then create them with correct owner and privileges

    Kafka

    /etc/kafka/conf/kafka_client_jaas.conf
    /etc/kafka/conf/kafka_jaas.conf

    Zookeeper

    /etc/zookeeper/conf/zookeeper_jaas.conf
    /etc/zookeeper/conf/zookeeper_client_jaas.conf

    After updating these files restart the zk and Kafka and retry

    HTH

    Translate
    ururu
    ururuAuthor
    Gold ⭐️
    January 23, 2019

    @Geoffrey Shelton Okot

    I compared files, yes their exists and the same.

    Translate
    Shelton
    Shelton
    Gold ⭐️⭐️⭐️
    January 23, 2019

    @ Ruslan Fialkovsky

    Logon to Zk on DEV and PROD Check whether you have child znodes under

    ls /config/topics

    Validate that you have Kerberos entries in /etc/nifi/conf/nifi_jaas.conf

    example

    NiFiClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true keyTab="/etc/security/keytabs/nifi.service.keytab"
    storeKey=true
    useTicketCache=false
    principal="nifi/{kdc_host}@REALM";
    };
    RegistryClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true keyTab="/etc/security/keytabs/nifi.service.keytab"
    storeKey=true
    useTicketCache=false
    principal="nifi/{kdc_host}@REALM";

    Please revert

    Translate
    ururu
    ururuAuthor
    Gold ⭐️
    January 23, 2019

    @Geoffrey Shelton Okot

    Yes i have

    ls /config/topics
    [test1]

    NiFiClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/nifi.service.keytab"
    storeKey=true
    useTicketCache=false
    principal="nifi/host@RAIFFEISEN.RU";
    };
    RegistryClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/etc/security/keytabs/nifi.service.keytab"
    storeKey=true
    useTicketCache=false
    principal="nifi/host@RAIFFEISEN.RU";

    Translate
    Shelton
    Shelton
    Gold ⭐️⭐️⭐️
    January 24, 2019

    @Ruslan Fialkovsky

    I think the topics created before changing the ACL's don't inherit the permissions. From the znode entry, I see you have one topic called test.

    To validate can you create a new topic and ensure you change them in your Nifi flow files and retest! Normally if you run rmr in zookeeper CLI the entry deleted should be created but I am not sure for Kafka topics.

    Please revert

    Translate
    ururu
    ururuAuthor
    Gold ⭐️
    February 5, 2019

    @Geoffrey Shelton Okot

    Thank you, you are right, the problem was really in zookeeper's acl.

    I copied everything in "ZooKeeper directory" from Test cluster to Dev cluster and that was help. But i don't know what exactly permission affected it. Is something way to get list all acl permission by Zookeeper? I would like to compare it with all acl from both cluster.

    101381-capture.jpg

    Translate
    Terms & ConditionsAccessibility statement