Nifi consuming data from Kafka using consumeKafka_0_10

[#1]

I am using NiFi to read data from apache Kafka version 0.10. In using the processor, it asks for my kafka broker ip's instead of zookeeper connects. Since brokers can keep changing in my usecase, can I mention the zookeeper connects to NiFi as was in getKafka processor?

[#2]

You can specify a list of brokers. The main difference in Kafka from before was client-managed offsets, i.e. ZK was no longer on the read path for a consumer.

Also, why would you expect brokers to keep changing?

[#3]

From my understanding zookeeper knows all the kafka brokers that are currently present in the cluster. So when the consumer/consumer group connects, it can get to know which brokers to connect to. Correct me if I am wrong. In my usecase, kafka cluster is running in an cloud environment and kafka brokers can get added to the cluster or removed based on needs, dynamically. This means zookeepers are the only machines that are here to stay. That is why giving zookeeper connects makes my life easier in this scenario :). Is it possible?

[#4]

I would be very cautious about that statement. In a highly available environment Kafka brokers don't just come and go. They are part of a replication group and persisted topic data must follow or be replicated to another host if a node is gone.

[#5]

When you say list of brokers, you mean the kafka brokers and not zookeeper right?

[#6 · Accepted solution]

What Andrew is saying is that Kafka has moved away from using ZooKeeper for a lot of the consumer side things. Starting in 0.9 they store offsets directly in Kafka and have a new way of doing group management. You can find a lot of articles about it, one of them http://www.jesse-anderson.com/2016/04/kafka-0-9-0-changes-for-developers/

You'll see all examples for 0.9 and 0.10 consumers take "bootstrap servers" (i.e. brokers) rather than ZK connection string.

[#7]

So I need to use kafka brokers. Is there a way for using zookeepers?

[#8]

NiFi's Kafka processors use the KafkaConsumer [1] provided by the Apache Kafka client library, and that consumer uses the "bootstrap.servers" for versions 0.9 and 0.10 of the client library, so there is no way to use ZooKeeper.

(https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.htm...