HDP 2.3: Kafka MirrorMaker hangs.

[#1]

I'm using Kafka and MirrorMaker in HDP 2.3.4.

The MirrorMaker sometimes stops with following error logs:

[2016-08-02 19:27:59,832] ERROR Error UNKNOWN_MEMBER_ID occurred while committing offsets for group mirrormaker_cc1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2016-08-02 19:27:59,833] ERROR User provided listener kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer failed on partition revocation:  (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:546)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:487)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:671)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:650)
        at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
        at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
        at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352)
        at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:941)
        at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.commit(MirrorMaker.scala:546)
        at kafka.tools.MirrorMaker$.commitOffsets(MirrorMaker.scala:340)
        at kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer.onPartitionsRevoked(MirrorMaker.scala:557)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:244)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:208)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:305)
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:864)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:827)
        at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:527)
        at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:395)

It is similar to the logs described in KAFKA-3409.

This problem is already fixed in Kafka upstream, but any HDP 2.3.x releases do not include the fix for it. So I deployed a jar that is rebuilt with the following 3 commits on top of HDP-2.3.4.0-tag.

- https://github.com/apache/kafka/commit/8f302c83c3be0f5264a07dae73287dec81a0198d

- https://github.com/apache/kafka/commit/c453dd607a5ce88463ee47e282c58c28427894ad

- https://github.com/apache/kafka/commit/2835e73000a44c9586d085275b62eeeb2ce1fcec

Is there any plan to fix this problem in HDP 2.3.x releases?

[#2]

HDP 2.3.6 and HDP 2.4.2 are pretty much the same except for Hive 1.2.1 and Spark 1.6. Kafka mirror maker is now supported and perhaps your issue is resolved in 2.3.6 as well. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_HDP_RelNotes/content/new_features.html

I think however there's a documentation issue in the release notes that contradicts this statement and I will follow up. Please try upgrading to 2.3.6 or test this on another cluster with 2.3.6 to confirm.

[#3]

@Shun Takebayashi please upgrade to HDP 2.3.6 as that has fixes for MirrorMaker and support from HWX.

[#4]

Same error is reproduced in HDP 2.3.6.

KAFKA-3409 patches commitOffsets(), but 2.3.6 doesn't have that patch. (https://github.com/hortonworks/kafka-release/blob/HDP-2.3.6.2-tag/core/src/main/scala/kafka/tools/Mi...)

Even HDP 2.4.2 doesn't have the said patch. (https://github.com/hortonworks/kafka-release/blob/HDP-2.4.2.16-tag/core/src/main/scala/kafka/tools/M...)

Please look into this and let us know the possibilities of backporting this patch. Thanks

[#5 · Accepted solution]

@Shun Takebayashi We will be backporting it for next maint release fro 2.3 and 2.4