I'm using Kafka and MirrorMaker in HDP 2.3.4. The MirrorMaker sometimes stops with following error logs: [2016-08-02 19:27:59,832] ERROR Error UNKNOWN_MEMBER_ID occurred while committing offsets for group mirrormaker_cc1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2016-08-02 19:27:59,833] ERROR User provided listener kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer failed on partition revocation: (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:546) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:487) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:671) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:650) at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352) at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:941) at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.commit(MirrorMaker.scala:546) at kafka.tools.MirrorMaker$.commitOffsets(MirrorMaker.scala:340) at kafka.tools.MirrorMaker$InternalRebalanceListenerForNewConsumer.onPartitionsRevoked(MirrorMaker.scala:557) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:244) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:208) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:305) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:864) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:827) at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:527) at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:395) It is similar to the logs described in KAFKA-3409. This problem is already fixed in Kafka upstream, but any HDP 2.3.x releases do not include the fix for it. So I deployed a jar that is rebuilt with the following 3 commits on top of HDP-2.3.4.0-tag. - https://github.com/apache/kafka/commit/8f302c83c3be0f5264a07dae73287dec81a0198d - https://github.com/apache/kafka/commit/c453dd607a5ce88463ee47e282c58c28427894ad - https://github.com/apache/kafka/commit/2835e73000a44c9586d085275b62eeeb2ce1fcec Is there any plan to fix this problem in HDP 2.3.x releases? ... View more