Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.3: Kafka MirrorMaker hangs.

Solved Go to solution
Highlighted

HDP 2.3: Kafka MirrorMaker hangs.

New Contributor

I'm using Kafka and MirrorMaker in HDP 2.3.4.

The MirrorMaker sometimes stops with following error logs:

[2016-08-02 19:27:59,832] ERROR Error UNKNOWN_MEMBER_ID occurred while committing offsets for group mirrormaker_cc1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2016-08-02 19:27:59,833] ERROR User provided listener kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer failed on partition revocation:  (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:546)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:487)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:671)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:650)
        at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
        at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
        at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352)
        at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:941)
        at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.commit(MirrorMaker.scala:546)
        at kafka.tools.MirrorMaker$.commitOffsets(MirrorMaker.scala:340)
        at kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer.onPartitionsRevoked(MirrorMaker.scala:557)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:244)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:208)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:305)
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:864)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:827)
        at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:527)
        at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:395)

It is similar to the logs described in KAFKA-3409.

This problem is already fixed in Kafka upstream, but any HDP 2.3.x releases do not include the fix for it. So I deployed a jar that is rebuilt with the following 3 commits on top of HDP-2.3.4.0-tag.

- https://github.com/apache/kafka/commit/8f302c83c3be0f5264a07dae73287dec81a0198d

- https://github.com/apache/kafka/commit/c453dd607a5ce88463ee47e282c58c28427894ad

- https://github.com/apache/kafka/commit/2835e73000a44c9586d085275b62eeeb2ce1fcec

Is there any plan to fix this problem in HDP 2.3.x releases?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: HDP 2.3: Kafka MirrorMaker hangs.

@Shun Takebayashi We will be backporting it for next maint release fro 2.3 and 2.4

View solution in original post

4 REPLIES 4
Highlighted

Re: HDP 2.3: Kafka MirrorMaker hangs.

Mentor

HDP 2.3.6 and HDP 2.4.2 are pretty much the same except for Hive 1.2.1 and Spark 1.6. Kafka mirror maker is now supported and perhaps your issue is resolved in 2.3.6 as well. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_HDP_RelNotes/content/new_features.html

I think however there's a documentation issue in the release notes that contradicts this statement and I will follow up. Please try upgrading to 2.3.6 or test this on another cluster with 2.3.6 to confirm.

Highlighted

Re: HDP 2.3: Kafka MirrorMaker hangs.

Mentor

@Shun Takebayashi please upgrade to HDP 2.3.6 as that has fixes for MirrorMaker and support from HWX.

Highlighted

Re: HDP 2.3: Kafka MirrorMaker hangs.

New Contributor

Same error is reproduced in HDP 2.3.6.

KAFKA-3409 patches commitOffsets(), but 2.3.6 doesn't have that patch. (https://github.com/hortonworks/kafka-release/blob/HDP-2.3.6.2-tag/core/src/main/scala/kafka/tools/Mi...)

Even HDP 2.4.2 doesn't have the said patch. (https://github.com/hortonworks/kafka-release/blob/HDP-2.4.2.16-tag/core/src/main/scala/kafka/tools/M...)

Please look into this and let us know the possibilities of backporting this patch. Thanks

Highlighted

Re: HDP 2.3: Kafka MirrorMaker hangs.

@Shun Takebayashi We will be backporting it for next maint release fro 2.3 and 2.4

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here