Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP 2.3: Kafka MirrorMaker hangs.

avatar
Explorer

I'm using Kafka and MirrorMaker in HDP 2.3.4.

The MirrorMaker sometimes stops with following error logs:

[2016-08-02 19:27:59,832] ERROR Error UNKNOWN_MEMBER_ID occurred while committing offsets for group mirrormaker_cc1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2016-08-02 19:27:59,833] ERROR User provided listener kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer failed on partition revocation:  (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:546)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:487)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:671)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:650)
        at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
        at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
        at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352)
        at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:941)
        at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.commit(MirrorMaker.scala:546)
        at kafka.tools.MirrorMaker$.commitOffsets(MirrorMaker.scala:340)
        at kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer.onPartitionsRevoked(MirrorMaker.scala:557)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:244)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:208)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:305)
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:864)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:827)
        at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:527)
        at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:395)

It is similar to the logs described in KAFKA-3409.

This problem is already fixed in Kafka upstream, but any HDP 2.3.x releases do not include the fix for it. So I deployed a jar that is rebuilt with the following 3 commits on top of HDP-2.3.4.0-tag.

- https://github.com/apache/kafka/commit/8f302c83c3be0f5264a07dae73287dec81a0198d

- https://github.com/apache/kafka/commit/c453dd607a5ce88463ee47e282c58c28427894ad

- https://github.com/apache/kafka/commit/2835e73000a44c9586d085275b62eeeb2ce1fcec

Is there any plan to fix this problem in HDP 2.3.x releases?

1 ACCEPTED SOLUTION

avatar

@Shun Takebayashi We will be backporting it for next maint release fro 2.3 and 2.4

View solution in original post

4 REPLIES 4

avatar
Master Mentor

HDP 2.3.6 and HDP 2.4.2 are pretty much the same except for Hive 1.2.1 and Spark 1.6. Kafka mirror maker is now supported and perhaps your issue is resolved in 2.3.6 as well. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_HDP_RelNotes/content/new_features.html

I think however there's a documentation issue in the release notes that contradicts this statement and I will follow up. Please try upgrading to 2.3.6 or test this on another cluster with 2.3.6 to confirm.

avatar
Master Mentor

@Shun Takebayashi please upgrade to HDP 2.3.6 as that has fixes for MirrorMaker and support from HWX.

avatar
Explorer

Same error is reproduced in HDP 2.3.6.

KAFKA-3409 patches commitOffsets(), but 2.3.6 doesn't have that patch. (https://github.com/hortonworks/kafka-release/blob/HDP-2.3.6.2-tag/core/src/main/scala/kafka/tools/Mi...)

Even HDP 2.4.2 doesn't have the said patch. (https://github.com/hortonworks/kafka-release/blob/HDP-2.4.2.16-tag/core/src/main/scala/kafka/tools/M...)

Please look into this and let us know the possibilities of backporting this patch. Thanks

avatar

@Shun Takebayashi We will be backporting it for next maint release fro 2.3 and 2.4