- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
HDP 2.3: Kafka MirrorMaker hangs.
Created 08-16-2016 05:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm using Kafka and MirrorMaker in HDP 2.3.4.
The MirrorMaker sometimes stops with following error logs:
[2016-08-02 19:27:59,832] ERROR Error UNKNOWN_MEMBER_ID occurred while committing offsets for group mirrormaker_cc1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2016-08-02 19:27:59,833] ERROR User provided listener kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer failed on partition revocation: (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:546) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:487) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:671) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:650) at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352) at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:941) at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.commit(MirrorMaker.scala:546) at kafka.tools.MirrorMaker$.commitOffsets(MirrorMaker.scala:340) at kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer.onPartitionsRevoked(MirrorMaker.scala:557) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:244) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:208) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:305) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:864) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:827) at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:527) at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:395)
It is similar to the logs described in KAFKA-3409.
This problem is already fixed in Kafka upstream, but any HDP 2.3.x releases do not include the fix for it. So I deployed a jar that is rebuilt with the following 3 commits on top of HDP-2.3.4.0-tag.
- https://github.com/apache/kafka/commit/8f302c83c3be0f5264a07dae73287dec81a0198d
- https://github.com/apache/kafka/commit/c453dd607a5ce88463ee47e282c58c28427894ad
- https://github.com/apache/kafka/commit/2835e73000a44c9586d085275b62eeeb2ce1fcec
Is there any plan to fix this problem in HDP 2.3.x releases?
Created 08-23-2016 02:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Shun Takebayashi We will be backporting it for next maint release fro 2.3 and 2.4
Created 08-16-2016 11:33 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
HDP 2.3.6 and HDP 2.4.2 are pretty much the same except for Hive 1.2.1 and Spark 1.6. Kafka mirror maker is now supported and perhaps your issue is resolved in 2.3.6 as well. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_HDP_RelNotes/content/new_features.html
I think however there's a documentation issue in the release notes that contradicts this statement and I will follow up. Please try upgrading to 2.3.6 or test this on another cluster with 2.3.6 to confirm.
Created 08-17-2016 06:48 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Shun Takebayashi please upgrade to HDP 2.3.6 as that has fixes for MirrorMaker and support from HWX.
Created 08-23-2016 10:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same error is reproduced in HDP 2.3.6.
KAFKA-3409 patches commitOffsets(), but 2.3.6 doesn't have that patch. (https://github.com/hortonworks/kafka-release/blob/HDP-2.3.6.2-tag/core/src/main/scala/kafka/tools/Mi...)
Even HDP 2.4.2 doesn't have the said patch. (https://github.com/hortonworks/kafka-release/blob/HDP-2.4.2.16-tag/core/src/main/scala/kafka/tools/M...)
Please look into this and let us know the possibilities of backporting this patch. Thanks
Created 08-23-2016 02:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Shun Takebayashi We will be backporting it for next maint release fro 2.3 and 2.4
