Nifi ConsumeKafka_0_10 error

[#1]

Hi All,

I got an error while using the ConsumeKafka_0_10 processor . Below is the complete error from nifi-app.log

2018-02-16 11:47:11,465 WARN [Timer-Driven Process Thread-2] o.a.n.p.kafka.pubsub.ConsumeKafka_0_10 ConsumeKafka_0_10[id=b846165f-115a-1161-cadd-815bbb3a45dd] Was interrupted while trying to communicate with Kafka with lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@6cb8afba. Will roll back session and discard any partially received data.

bootstrap.servers = [xxx-xxxkafka-xxxz.xxx.xx.local:9093]

key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]

value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

The last time I saw this error I restarted the nifi and it was gone but it did not work this time. There hasn't been any change on the kafka side.

I'm using nifi 1.4.0 along with kafka_2.11-0.11.0.1 to consume kafka records , please let me know if any one knows the root cause of this error.

[#2]

When did this error occured? Did something happened? For example you made a change in the consumer processor topic or group id?

[#3]

I did not make any changes to the consumer process or the group id. The offset reset option is set to latest. There were also no changes made to the topic on the consumer side at least. The topic which in this case is a name of an Oracle table is generated in kafka when the table is created in Oracle. I am pretty sure there were no changes made in the configuration on goldengate as well. As the team that manages those server have made changes to Goldengate configuration in the past with no disruption to workflows in Nifi. The server on which Goldengate is installed is different from the one that hosts Nifi , Can a network disruption cause this error?

[#4 · Accepted solution]

Network issues can certainly be a factor. However, you might also want to ensure you use the precise Kafka client for the given Kafka broker version. Since you're on Kafka 0.11 you might want NiFi 1.5.0 or HDF 3.1.0 which has support for that directly in ConsumeKafka_0_11