Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)

Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)

New Contributor
Im in the process of upgrading our kafka cluster

Brokers are in version 10.1.1

and consumer/producers 0.8.2-beta

Here is the the config for brokers:

###
### This file is managed by Puppet.
###


# See http://kafka.apache.org/documentation.html#brokerconfigs for default values.


# The id of the broker. This must be set to a unique integer for each broker.
broker.id=42


# The port the socket server listens on
port=9092


# A comma seperated list of directories under which to store log files
log.dirs=/kafka1/datalog,/kafka2/datalog,/kafka3/datalog,/kafka4/datalog,/kafka5/datalog,/kafka6/datalog,/kafka7/datalog,/kafka8/datalog,/kafka9/datalog,/kafka10/datalog


# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.


zookeeper.connect=zookeeper1-repl:2181,zookeeper2-repl:2181,zookeeper3-repl:2181,zookeeper4-repl:2181,zookeeper5-repl:2181/replication/kafka


# Additional configuration options may follow here
auto.leader.rebalance.enable=true
delete.topic.enable=true
socket.receive.buffer.bytes=1048576
socket.send.buffer.bytes=1048576
default.replication.factor=2
auto.create.topics.enable=true
num.partitions=1
num.network.threads=8
num.io.threads=40
log.retention.hours=1
log.roll.hours=1
num.replica.fetchers=8
zookeeper.connection.timeout.ms=30000
zookeeper.session.timeout.ms=30000
inter.broker.protocol.version=0.8.2
log.message.format.version=0.8.2

Here the config for consumers :

bootstrap.servers=app454.sjc2.com:9092,app455.sjc2.com:9092,app456.sjc2com:9092,app457.sjc2.com:9092,app458.sjc2.com:9092,app459.sjc2.com:9092


group.id=MirrorMaker_hkg1
enable.auto.commit=true
auto.commit.interval.ms=60000
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

The Producer

bootstrap.servers=app202.hkg1.com:9092,app203.hkg1.com:9092,app204.hkg1.com:9092,app205.hkg1.com:9092,app206.hkg1.com:9092,app207.hkg1.com:9092
acks=0
compression.type=gzip
producer.type=async

We are using mirrormaker

So brokers start fine, however when I start mirrormaker it fails with the following error:

[[A[2017-06-08 15:25:38,068] INFO Starting mirror maker (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:38,139] INFO Property acks is overridden to 0 - data loss or message reordering is possible. (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:38,179] INFO ProducerConfig values:

	acks = 0

	batch.size = 16384

	block.on.buffer.full = false

	bootstrap.servers = [app202.hkg1.mytest.com.com:9092, app203.hkg1.mytest.com.com:9092, app204.hkg1.mytest.com.com:9092, app205.hkg1.mytest.com.com:9092, app206.hkg1.mytest.com.com:9092, app207.hkg1.mytest.com.com:9092]

	buffer.memory = 33554432

	client.id =

	compression.type = gzip

	connections.max.idle.ms = 540000

	interceptor.classes = null

	key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer

	linger.ms = 0

	max.block.ms = 9223372036854775807

	max.in.flight.requests.per.connection = 1

	max.request.size = 1048576

	metadata.fetch.timeout.ms = 60000

	metadata.max.age.ms = 300000

	metric.reporters = []

	metrics.num.samples = 2

	metrics.sample.window.ms = 30000

	partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner

	receive.buffer.bytes = 32768

	reconnect.backoff.ms = 50

	request.timeout.ms = 30000

	retries = 2147483647

	retry.backoff.ms = 100

	sasl.jaas.config = null

	sasl.kerberos.kinit.cmd = /usr/bin/kinit

	sasl.kerberos.min.time.before.relogin = 60000

	sasl.kerberos.service.name = null

	sasl.kerberos.ticket.renew.jitter = 0.05

	sasl.kerberos.ticket.renew.window.factor = 0.8

	sasl.mechanism = GSSAPI

	security.protocol = PLAINTEXT

	send.buffer.bytes = 131072

	ssl.cipher.suites = null

	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

	ssl.endpoint.identification.algorithm = null

	ssl.key.password = null

	ssl.keymanager.algorithm = SunX509

	ssl.keystore.location = null

	ssl.keystore.password = null

	ssl.keystore.type = JKS

	ssl.protocol = TLS

	ssl.provider = null

	ssl.secure.random.implementation = null

	ssl.trustmanager.algorithm = PKIX

	ssl.truststore.location = null

	ssl.truststore.password = null

	ssl.truststore.type = JKS

	timeout.ms = 30000

	value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer

 (org.apache.kafka.clients.producer.ProducerConfig)

[2017-06-08 15:25:38,212] DEBUG Added sensor with name bufferpool-wait-time (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,217] DEBUG Added sensor with name buffer-exhausted-records (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,221] DEBUG Updated cluster metadata version 1 to Cluster(id = null, nodes = [app204.hkg1.mytest.com.com:9092 (id: -3 rack: null), app205.hkg1.mytest.com.com:9092 (id: -4 rack: null), app207.hkg1.mytest.com.com:9092 (id: -6 rack: null), app203.hkg1.mytest.com.com:9092 (id: -2 rack: null), app202.hkg1.mytest.com.com:9092 (id: -1 rack: null), app206.hkg1.mytest.com.com:9092 (id: -5 rack: null)], partitions = []) (org.apache.kafka.clients.Metadata)

[2017-06-08 15:25:38,238] DEBUG Added sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,238] DEBUG Added sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,239] DEBUG Added sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,239] DEBUG Added sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,242] DEBUG Added sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,242] DEBUG Added sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,243] DEBUG Added sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,253] DEBUG Added sensor with name batch-size (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,254] DEBUG Added sensor with name compression-rate (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,254] DEBUG Added sensor with name queue-time (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,255] DEBUG Added sensor with name request-time (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,256] DEBUG Added sensor with name produce-throttle-time (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,256] DEBUG Added sensor with name records-per-request (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,257] DEBUG Added sensor with name record-retries (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,257] DEBUG Added sensor with name errors (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,258] DEBUG Added sensor with name record-size-max (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,262] DEBUG Starting Kafka producer I/O thread. (org.apache.kafka.clients.producer.internals.Sender)

[2017-06-08 15:25:38,265] INFO Kafka version : 0.10.2.1 (org.apache.kafka.common.utils.AppInfoParser)

[2017-06-08 15:25:38,265] INFO Kafka commitId : e89bffd6b2eff799 (org.apache.kafka.common.utils.AppInfoParser)

[2017-06-08 15:25:38,266] DEBUG Kafka producer started (org.apache.kafka.clients.producer.KafkaProducer)

[2017-06-08 15:25:38,267] INFO Property enable.auto.commit is overridden to true - data loss or message reordering is possible. (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:38,286] INFO ConsumerConfig values:

	auto.commit.interval.ms = 60000

	auto.offset.reset = latest

	bootstrap.servers = [app454.sjc2.mytest.com.com:9092, app455.sjc2.mytest.com.com:9092, app456.sjc2.mytest.com.com:9092, app457.sjc2.mytest.com.com:9092, app458.sjc2.mytest.com.com:9092, app459.sjc2.mytest.com.com:9092]

	check.crcs = true

	client.id = MirrorMaker_hkg1-0

	connections.max.idle.ms = 540000

	enable.auto.commit = true

	exclude.internal.topics = true

	fetch.max.bytes = 52428800

	fetch.max.wait.ms = 500

	fetch.min.bytes = 1

	group.id = MirrorMaker_hkg1

	heartbeat.interval.ms = 3000

	interceptor.classes = null

	key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

	max.partition.fetch.bytes = 1048576

	max.poll.interval.ms = 300000

	max.poll.records = 500

	metadata.max.age.ms = 300000

	metric.reporters = []

	metrics.num.samples = 2

	metrics.recording.level = INFO

	metrics.sample.window.ms = 30000

	partition.assignment.strategy = [org.apache.kafka.clients.consumer.RoundRobinAssignor]

	receive.buffer.bytes = 65536

	reconnect.backoff.ms = 50

	request.timeout.ms = 305000

	retry.backoff.ms = 100

	sasl.jaas.config = null

	sasl.kerberos.kinit.cmd = /usr/bin/kinit

	sasl.kerberos.min.time.before.relogin = 60000

	sasl.kerberos.service.name = null

	sasl.kerberos.ticket.renew.jitter = 0.05

	sasl.kerberos.ticket.renew.window.factor = 0.8

	sasl.mechanism = GSSAPI

	security.protocol = PLAINTEXT

	send.buffer.bytes = 131072

	session.timeout.ms = 10000

	ssl.cipher.suites = null

	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

	ssl.endpoint.identification.algorithm = null

	ssl.key.password = null

	ssl.keymanager.algorithm = SunX509

	ssl.keystore.location = null

	ssl.keystore.password = null

	ssl.keystore.type = JKS

	ssl.protocol = TLS

	ssl.provider = null

	ssl.secure.random.implementation = null

	ssl.trustmanager.algorithm = PKIX

	ssl.truststore.location = null

	ssl.truststore.password = null

	ssl.truststore.type = JKS

	value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

 (org.apache.kafka.clients.consumer.ConsumerConfig)

[2017-06-08 15:25:38,286] DEBUG Starting the Kafka consumer (org.apache.kafka.clients.consumer.KafkaConsumer)

[2017-06-08 15:25:38,290] DEBUG Updated cluster metadata version 1 to Cluster(id = null, nodes = [app455.sjc2.mytest.com.com:9092 (id: -2 rack: null), app456.sjc2.mytest.com.com:9092 (id: -3 rack: null), app459.sjc2.mytest.com.com:9092 (id: -6 rack: null), app454.sjc2.mytest.com.com:9092 (id: -1 rack: null), app457.sjc2.mytest.com.com:9092 (id: -4 rack: null), app458.sjc2.mytest.com.com:9092 (id: -5 rack: null)], partitions = []) (org.apache.kafka.clients.Metadata)

[2017-06-08 15:25:38,291] DEBUG Added sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,291] DEBUG Added sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,291] DEBUG Added sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,292] DEBUG Added sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,292] DEBUG Added sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,293] DEBUG Added sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,293] DEBUG Added sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,319] DEBUG Added sensor with name heartbeat-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,319] DEBUG Added sensor with name join-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,320] DEBUG Added sensor with name sync-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,324] DEBUG Added sensor with name commit-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,338] DEBUG Added sensor with name bytes-fetched (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,338] DEBUG Added sensor with name records-fetched (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,339] DEBUG Added sensor with name fetch-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,339] DEBUG Added sensor with name records-lag (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,339] DEBUG Added sensor with name fetch-throttle-time (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,340] INFO Kafka version : 0.10.2.1 (org.apache.kafka.common.utils.AppInfoParser)

[2017-06-08 15:25:38,340] INFO Kafka commitId : e89bffd6b2eff799 (org.apache.kafka.common.utils.AppInfoParser)

[2017-06-08 15:25:38,340] DEBUG Kafka consumer created (org.apache.kafka.clients.consumer.KafkaConsumer)

[2017-06-08 15:25:38,342] INFO ConsumerConfig values:

	auto.commit.interval.ms = 60000

	auto.offset.reset = latest

	bootstrap.servers = [app454.sjc2.mytest.com.com:9092, app455.sjc2.mytest.com.com:9092, app456.sjc2.mytest.com.com:9092, app457.sjc2.mytest.com.com:9092, app458.sjc2.mytest.com.com:9092, app459.sjc2.mytest.com.com:9092]

	check.crcs = true

	client.id = MirrorMaker_hkg1-1

	connections.max.idle.ms = 540000

	enable.auto.commit = true

	exclude.internal.topics = true

	fetch.max.bytes = 52428800

	fetch.max.wait.ms = 500

	fetch.min.bytes = 1

	group.id = MirrorMaker_hkg1

	heartbeat.interval.ms = 3000

	interceptor.classes = null

	key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

	max.partition.fetch.bytes = 1048576

	max.poll.interval.ms = 300000

	max.poll.records = 500

	metadata.max.age.ms = 300000

	metric.reporters = []

	metrics.num.samples = 2

	metrics.recording.level = INFO

	metrics.sample.window.ms = 30000

	partition.assignment.strategy = [org.apache.kafka.clients.consumer.RoundRobinAssignor]

	receive.buffer.bytes = 65536

	reconnect.backoff.ms = 50

	request.timeout.ms = 305000

	retry.backoff.ms = 100

	sasl.jaas.config = null

	sasl.kerberos.kinit.cmd = /usr/bin/kinit

	sasl.kerberos.min.time.before.relogin = 60000

	sasl.kerberos.service.name = null

	sasl.kerberos.ticket.renew.jitter = 0.05

	sasl.kerberos.ticket.renew.window.factor = 0.8

	sasl.mechanism = GSSAPI

	security.protocol = PLAINTEXT

	send.buffer.bytes = 131072

	session.timeout.ms = 10000

	ssl.cipher.suites = null

	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

	ssl.endpoint.identification.algorithm = null

	ssl.key.password = null

	ssl.keymanager.algorithm = SunX509

	ssl.keystore.location = null

	ssl.keystore.password = null

	ssl.keystore.type = JKS

	ssl.protocol = TLS

	ssl.provider = null

	ssl.secure.random.implementation = null

	ssl.trustmanager.algorithm = PKIX

	ssl.truststore.location = null

	ssl.truststore.password = null

	ssl.truststore.type = JKS

	value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

 (org.apache.kafka.clients.consumer.ConsumerConfig)

[2017-06-08 15:25:38,342] DEBUG Starting the Kafka consumer (org.apache.kafka.clients.consumer.KafkaConsumer)

[2017-06-08 15:25:38,345] DEBUG Updated cluster metadata version 1 to Cluster(id = null, nodes = [app455.sjc2.mytest.com.com:9092 (id: -2 rack: null), app457.sjc2.mytest.com.com:9092 (id: -4 rack: null), app454.sjc2.mytest.com.com:9092 (id: -1 rack: null), app459.sjc2.mytest.com.com:9092 (id: -6 rack: null), app456.sjc2.mytest.com.com:9092 (id: -3 rack: null), app458.sjc2.mytest.com.com:9092 (id: -5 rack: null)], partitions = []) (org.apache.kafka.clients.Metadata)

[2017-06-08 15:25:38,346] DEBUG Added sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,346] DEBUG Added sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,346] DEBUG Added sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,347] DEBUG Added sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,348] DEBUG Added sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,348] DEBUG Added sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,349] DEBUG Added sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,350] DEBUG Added sensor with name heartbeat-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,350] DEBUG Added sensor with name join-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,351] DEBUG Added sensor with name sync-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,352] DEBUG Added sensor with name commit-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,353] DEBUG Added sensor with name bytes-fetched (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,353] DEBUG Added sensor with name records-fetched (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,354] DEBUG Added sensor with name fetch-latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,354] DEBUG Added sensor with name records-lag (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,355] DEBUG Added sensor with name fetch-throttle-time (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,355] INFO Kafka version : 0.10.2.1 (org.apache.kafka.common.utils.AppInfoParser)

[2017-06-08 15:25:38,355] INFO Kafka commitId : e89bffd6b2eff799 (org.apache.kafka.common.utils.AppInfoParser)

[2017-06-08 15:25:38,356] DEBUG Kafka consumer created (org.apache.kafka.clients.consumer.KafkaConsumer)

[2017-06-08 15:25:38,366] INFO [mirrormaker-thread-1] Starting mirror maker thread mirrormaker-thread-1 (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:38,366] INFO [mirrormaker-thread-0] Starting mirror maker thread mirrormaker-thread-0 (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:38,367] DEBUG Initiating new consumer (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:38,367] DEBUG Initiating new consumer (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:38,374] DEBUG Subscribed to pattern: REPL-ams1-global|REPL-atl1-global|REPL-sjc2-global|REPL-ams1-global-PN_HXIDMAP_.*|REPL-atl1-global-PN_HXIDMAP_.*|REPL-sjc2-global-PN_HXIDMAP_.*|REPL-ams1-global-PN_HXCONTEXTUALV2_.*|REPL-atl1-global-PN_HXCONTEXTUALV2_.*|REPL-sjc2-global-PN_HXCONTEXTUALV2_.* (org.apache.kafka.clients.consumer.KafkaConsumer)

[2017-06-08 15:25:38,374] DEBUG Sending GroupCoordinator request for group MirrorMaker_hkg1 to broker app458.sjc2.mytest.com.com:9092 (id: -5 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:38,377] DEBUG Subscribed to pattern: REPL-ams1-global|REPL-atl1-global|REPL-sjc2-global|REPL-ams1-global-PN_HXIDMAP_.*|REPL-atl1-global-PN_HXIDMAP_.*|REPL-sjc2-global-PN_HXIDMAP_.*|REPL-ams1-global-PN_HXCONTEXTUALV2_.*|REPL-atl1-global-PN_HXCONTEXTUALV2_.*|REPL-sjc2-global-PN_HXCONTEXTUALV2_.* (org.apache.kafka.clients.consumer.KafkaConsumer)

[2017-06-08 15:25:38,378] DEBUG Sending GroupCoordinator request for group MirrorMaker_hkg1 to broker app455.sjc2.mytest.com.com:9092 (id: -2 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:38,402] DEBUG Initiating connection to node -5 at app458.sjc2.mytest.com.com:9092. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,402] DEBUG Initiating connection to node -2 at app455.sjc2.mytest.com.com:9092. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,549] DEBUG Added sensor with name node--2.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,549] DEBUG Added sensor with name node--5.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,551] DEBUG Added sensor with name node--2.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,551] DEBUG Added sensor with name node--5.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,552] DEBUG Added sensor with name node--2.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,552] DEBUG Added sensor with name node--5.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,553] DEBUG Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2 (org.apache.kafka.common.network.Selector)

[2017-06-08 15:25:38,553] DEBUG Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -5 (org.apache.kafka.common.network.Selector)

[2017-06-08 15:25:38,553] DEBUG Completed connection to node -2.  Fetching API versions. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,553] DEBUG Completed connection to node -5.  Fetching API versions. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,553] DEBUG Initiating API versions fetch from node -2. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,553] DEBUG Initiating API versions fetch from node -5. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,572] DEBUG Initialize connection to node -4 for sending metadata request (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,572] DEBUG Initialize connection to node -5 for sending metadata request (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,572] DEBUG Initiating connection to node -4 at app457.sjc2.mytest.com.com:9092. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,572] DEBUG Initiating connection to node -5 at app458.sjc2.mytest.com.com:9092. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,715] DEBUG Added sensor with name node--5.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,716] DEBUG Added sensor with name node--5.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,716] DEBUG Added sensor with name node--5.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,717] DEBUG Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -5 (org.apache.kafka.common.network.Selector)

[2017-06-08 15:25:38,717] DEBUG Completed connection to node -5.  Fetching API versions. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,717] DEBUG Initiating API versions fetch from node -5. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,717] DEBUG Initialize connection to node -4 for sending metadata request (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,717] DEBUG Initiating connection to node -4 at app457.sjc2.mytest.com.com:9092. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,763] DEBUG Recorded API versions for node -2: (Produce(0): 0 to 2 [usable: 2], Fetch(1): 0 to 3 [usable: 3], Offsets(2): 0 to 1 [usable: 1], Metadata(3): 0 to 2 [usable: 2], LeaderAndIsr(4): 0 [usable: 0], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 2 [usable: 2], ControlledShutdown(7): 1 [usable: 1], OffsetCommit(8): 0 to 2 [usable: 2], OffsetFetch(9): 0 to 1 [usable: 1], GroupCoordinator(10): 0 [usable: 0], JoinGroup(11): 0 to 1 [usable: 1], Heartbeat(12): 0 [usable: 0], LeaveGroup(13): 0 [usable: 0], SyncGroup(14): 0 [usable: 0], DescribeGroups(15): 0 [usable: 0], ListGroups(16): 0 [usable: 0], SaslHandshake(17): 0 [usable: 0], ApiVersions(18): 0 [usable: 0], CreateTopics(19): 0 [usable: 0], DeleteTopics(20): 0 [usable: 0]) (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,763] DEBUG Recorded API versions for node -5: (Produce(0): 0 to 2 [usable: 2], Fetch(1): 0 to 3 [usable: 3], Offsets(2): 0 to 1 [usable: 1], Metadata(3): 0 to 2 [usable: 2], LeaderAndIsr(4): 0 [usable: 0], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 2 [usable: 2], ControlledShutdown(7): 1 [usable: 1], OffsetCommit(8): 0 to 2 [usable: 2], OffsetFetch(9): 0 to 1 [usable: 1], GroupCoordinator(10): 0 [usable: 0], JoinGroup(11): 0 to 1 [usable: 1], Heartbeat(12): 0 [usable: 0], LeaveGroup(13): 0 [usable: 0], SyncGroup(14): 0 [usable: 0], DescribeGroups(15): 0 [usable: 0], ListGroups(16): 0 [usable: 0], SaslHandshake(17): 0 [usable: 0], ApiVersions(18): 0 [usable: 0], CreateTopics(19): 0 [usable: 0], DeleteTopics(20): 0 [usable: 0]) (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,764] DEBUG Sending metadata request (type=MetadataRequest, topics=<ALL>) to node -2 (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,764] DEBUG Sending metadata request (type=MetadataRequest, topics=<ALL>) to node -5 (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,765] DEBUG Added sensor with name node--4.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,765] DEBUG Added sensor with name node--4.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,766] DEBUG Added sensor with name node--4.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,766] DEBUG Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -4 (org.apache.kafka.common.network.Selector)

[2017-06-08 15:25:38,766] DEBUG Completed connection to node -4.  Fetching API versions. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,766] DEBUG Initiating API versions fetch from node -4. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,867] DEBUG Added sensor with name node--4.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,869] DEBUG Added sensor with name node--4.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,869] DEBUG Added sensor with name node--4.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:38,871] DEBUG Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -4 (org.apache.kafka.common.network.Selector)

[2017-06-08 15:25:38,871] DEBUG Completed connection to node -4.  Fetching API versions. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,871] DEBUG Initiating API versions fetch from node -4. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,877] DEBUG Recorded API versions for node -5: (Produce(0): 0 to 2 [usable: 2], Fetch(1): 0 to 3 [usable: 3], Offsets(2): 0 to 1 [usable: 1], Metadata(3): 0 to 2 [usable: 2], LeaderAndIsr(4): 0 [usable: 0], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 2 [usable: 2], ControlledShutdown(7): 1 [usable: 1], OffsetCommit(8): 0 to 2 [usable: 2], OffsetFetch(9): 0 to 1 [usable: 1], GroupCoordinator(10): 0 [usable: 0], JoinGroup(11): 0 to 1 [usable: 1], Heartbeat(12): 0 [usable: 0], LeaveGroup(13): 0 [usable: 0], SyncGroup(14): 0 [usable: 0], DescribeGroups(15): 0 [usable: 0], ListGroups(16): 0 [usable: 0], SaslHandshake(17): 0 [usable: 0], ApiVersions(18): 0 [usable: 0], CreateTopics(19): 0 [usable: 0], DeleteTopics(20): 0 [usable: 0]) (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:38,928] DEBUG Recorded API versions for node -4: (Produce(0): 0 to 2 [usable: 2], Fetch(1): 0 to 3 [usable: 3], Offsets(2): 0 to 1 [usable: 1], Metadata(3): 0 to 2 [usable: 2], LeaderAndIsr(4): 0 [usable: 0], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 2 [usable: 2], ControlledShutdown(7): 1 [usable: 1], OffsetCommit(8): 0 to 2 [usable: 2], OffsetFetch(9): 0 to 1 [usable: 1], GroupCoordinator(10): 0 [usable: 0], JoinGroup(11): 0 to 1 [usable: 1], Heartbeat(12): 0 [usable: 0], LeaveGroup(13): 0 [usable: 0], SyncGroup(14): 0 [usable: 0], DescribeGroups(15): 0 [usable: 0], ListGroups(16): 0 [usable: 0], SaslHandshake(17): 0 [usable: 0], ApiVersions(18): 0 [usable: 0], CreateTopics(19): 0 [usable: 0], DeleteTopics(20): 0 [usable: 0]) (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:39,033] DEBUG Recorded API versions for node -4: (Produce(0): 0 to 2 [usable: 2], Fetch(1): 0 to 3 [usable: 3], Offsets(2): 0 to 1 [usable: 1], Metadata(3): 0 to 2 [usable: 2], LeaderAndIsr(4): 0 [usable: 0], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 2 [usable: 2], ControlledShutdown(7): 1 [usable: 1], OffsetCommit(8): 0 to 2 [usable: 2], OffsetFetch(9): 0 to 1 [usable: 1], GroupCoordinator(10): 0 [usable: 0], JoinGroup(11): 0 to 1 [usable: 1], Heartbeat(12): 0 [usable: 0], LeaveGroup(13): 0 [usable: 0], SyncGroup(14): 0 [usable: 0], DescribeGroups(15): 0 [usable: 0], ListGroups(16): 0 [usable: 0], SaslHandshake(17): 0 [usable: 0], ApiVersions(18): 0 [usable: 0], CreateTopics(19): 0 [usable: 0], DeleteTopics(20): 0 [usable: 0]) (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:42,826] DEBUG Updated cluster metadata version 2 to Cluster(id = ohAqP7DmTZagMoi5oxclAQ, nodes = [app459.sjc2.mytest.com.com:9092 (id: 35 rack: null), app455.sjc2.mytest.com.com:9092 (id: 31 rack: null), app454.sjc2.mytest.com.com:9092 (id: 30 rack: null), app457.sjc2.mytest.com.com:9092 (id: 33 rack: null), app456.sjc2.mytest.com.com:9092 (id: 32 rack: null), app458.sjc2.mytest.com.com:9092 (id: 34 rack: null)], partitions = [Partition(topic = REPL-atl1-global-PN_HXCONTEXTUALV2_490, partition = 0, leader = 35, replicas = [30,35], isr = [30,35]), Partition(topic = REPL-atl1-global-PN_HXCONTEXTUALV2_491, partition = 0, leader = 31, replicas = [34,31], isr = [31,34]), Partition(topic = REPL-atl1-global-PN_HXCONTEXTUALV2_492, partition = 0, leader = 32, replicas = [33,32], isr = [32,33]), Partition(topic = REPL-atl1-global-PN_HXCONTEXTUALV2_493, partition = 0, leader = 33, replicas = [33,35], isr = [33,35]), Partition(topic = REPL-atl1-global, partition = 72, leader = 31, replicas = [31,35], isr = [31,35]), Partition(topic = REPL-atl1-global-PN_HXCONTEXTUALV2_494, partition = 0, leader = 35, replicas = [30,35], isr = [30,35]), Partition(topic = REPL-atl1-global-PN_HXCONTEXTUALV2_495, partition = 0, leader = 33, replicas = [33,30], isr = [30,33]), Partition(topic = REPL-sjc2-global, partition = 136, leader = 30, replicas = [30,35], isr = [30,35]), Partition(topic = REPL-atl1-global-PN_HXCONTEXTUALV2_496, partition = 0, leader = 33, replicas = [33,30], isr = [30,33]), Partition(topic = REPL-atl1-global-PN_HXCONTEXTUALV2_497, partition = 0, leader = 32, replicas = [34,32], isr = [32,34]), (org.apache.kafka.clients.Metadata)

[2017-06-08 15:25:42,838] DEBUG Received GroupCoordinator response ClientResponse(receivedTimeMs=1496960742836, latencyMs=4439, disconnected=false, requestHeader={api_key=10,api_version=0,correlation_id=0,client_id=MirrorMaker_hkg1-0}, responseBody={error_code=0,coordinator={node_id=34,host=app458.sjc2.mytest.com.com,port=9092}}) for group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,841] DEBUG Received GroupCoordinator response ClientResponse(receivedTimeMs=1496960742841, latencyMs=4444, disconnected=false, requestHeader={api_key=10,api_version=0,correlation_id=0,client_id=MirrorMaker_hkg1-1}, responseBody={error_code=0,coordinator={node_id=34,host=app458.sjc2.mytest.com.com,port=9092}}) for group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,842] INFO Discovered coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null) for group MirrorMaker_hkg1. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,842] INFO Discovered coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null) for group MirrorMaker_hkg1. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,842] DEBUG Initiating connection to node 2147483613 at app458.sjc2.mytest.com.com:9092. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:42,842] DEBUG Initiating connection to node 2147483613 at app458.sjc2.mytest.com.com:9092. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:42,867] DEBUG Heartbeat thread for group MirrorMaker_hkg1 started (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,867] DEBUG Heartbeat thread for group MirrorMaker_hkg1 started (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,867] DEBUG Sending synchronous auto-commit of offsets {} for group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2017-06-08 15:25:42,867] DEBUG Sending synchronous auto-commit of offsets {} for group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2017-06-08 15:25:42,867] INFO Revoking previously assigned partitions [] for group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2017-06-08 15:25:42,867] INFO Revoking previously assigned partitions [] for group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2017-06-08 15:25:42,896] INFO (Re-)joining group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,898] INFO (Re-)joining group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,932] DEBUG Sending JoinGroup ((type: JoinGroupRequest, groupId=MirrorMaker_hkg1, sessionTimeout=10000, rebalanceTimeout=300000, memberId=, protocolType=consumer, groupProtocols=org.apache.kafka.common.requests.JoinGroupRequest$ProtocolMetadata@7931d9dc)) to coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,955] DEBUG Sending JoinGroup ((type: JoinGroupRequest, groupId=MirrorMaker_hkg1, sessionTimeout=10000, rebalanceTimeout=300000, memberId=, protocolType=consumer, groupProtocols=org.apache.kafka.common.requests.JoinGroupRequest$ProtocolMetadata@1c313f4e)) to coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:42,985] DEBUG Added sensor with name node-2147483613.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:42,985] DEBUG Added sensor with name node-2147483613.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:42,986] DEBUG Added sensor with name node-2147483613.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:42,986] DEBUG Added sensor with name node-2147483613.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:42,987] DEBUG Added sensor with name node-2147483613.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:42,987] DEBUG Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2147483613 (org.apache.kafka.common.network.Selector)

[2017-06-08 15:25:42,987] DEBUG Completed connection to node 2147483613.  Fetching API versions. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:42,987] DEBUG Initiating API versions fetch from node 2147483613. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:42,988] DEBUG Added sensor with name node-2147483613.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:42,988] DEBUG Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2147483613 (org.apache.kafka.common.network.Selector)

[2017-06-08 15:25:42,988] DEBUG Completed connection to node 2147483613.  Fetching API versions. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:42,988] DEBUG Initiating API versions fetch from node 2147483613. (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:43,140] DEBUG Recorded API versions for node 2147483613: (Produce(0): 0 to 2 [usable: 2], Fetch(1): 0 to 3 [usable: 3], Offsets(2): 0 to 1 [usable: 1], Metadata(3): 0 to 2 [usable: 2], LeaderAndIsr(4): 0 [usable: 0], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 2 [usable: 2], ControlledShutdown(7): 1 [usable: 1], OffsetCommit(8): 0 to 2 [usable: 2], OffsetFetch(9): 0 to 1 [usable: 1], GroupCoordinator(10): 0 [usable: 0], JoinGroup(11): 0 to 1 [usable: 1], Heartbeat(12): 0 [usable: 0], LeaveGroup(13): 0 [usable: 0], SyncGroup(14): 0 [usable: 0], DescribeGroups(15): 0 [usable: 0], ListGroups(16): 0 [usable: 0], SaslHandshake(17): 0 [usable: 0], ApiVersions(18): 0 [usable: 0], CreateTopics(19): 0 [usable: 0], DeleteTopics(20): 0 [usable: 0]) (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:43,140] DEBUG Recorded API versions for node 2147483613: (Produce(0): 0 to 2 [usable: 2], Fetch(1): 0 to 3 [usable: 3], Offsets(2): 0 to 1 [usable: 1], Metadata(3): 0 to 2 [usable: 2], LeaderAndIsr(4): 0 [usable: 0], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 2 [usable: 2], ControlledShutdown(7): 1 [usable: 1], OffsetCommit(8): 0 to 2 [usable: 2], OffsetFetch(9): 0 to 1 [usable: 1], GroupCoordinator(10): 0 [usable: 0], JoinGroup(11): 0 to 1 [usable: 1], Heartbeat(12): 0 [usable: 0], LeaveGroup(13): 0 [usable: 0], SyncGroup(14): 0 [usable: 0], DescribeGroups(15): 0 [usable: 0], ListGroups(16): 0 [usable: 0], SaslHandshake(17): 0 [usable: 0], ApiVersions(18): 0 [usable: 0], CreateTopics(19): 0 [usable: 0], DeleteTopics(20): 0 [usable: 0]) (org.apache.kafka.clients.NetworkClient)

[2017-06-08 15:25:44,955] DEBUG Received successful JoinGroup response for group MirrorMaker_hkg1: {error_code=0,generation_id=2,group_protocol=roundrobin,leader_id=MirrorMaker_hkg1-0-8c7d481e-8d2d-403c-acb0-c43486a1da59,member_id=MirrorMaker_hkg1-1-9e6708fc-1eb1-4a62-89d5-387d78865e9e,members=[]} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:44,956] DEBUG Sending follower SyncGroup for group MirrorMaker_hkg1 to coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null): (type=SyncGroupRequest, groupId=MirrorMaker_hkg1, generationId=2, memberId=MirrorMaker_hkg1-1-9e6708fc-1eb1-4a62-89d5-387d78865e9e, groupAssignment=) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:44,957] DEBUG Received successful JoinGroup response for group MirrorMaker_hkg1: {error_code=0,generation_id=2,group_protocol=roundrobin,leader_id=MirrorMaker_hkg1-0-8c7d481e-8d2d-403c-acb0-c43486a1da59,member_id=MirrorMaker_hkg1-0-3f369f11-3ce6-40df-bc40-ca817f01339a,members=[]} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:44,958] DEBUG Sending follower SyncGroup for group MirrorMaker_hkg1 to coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null): (type=SyncGroupRequest, groupId=MirrorMaker_hkg1, generationId=2, memberId=MirrorMaker_hkg1-0-3f369f11-3ce6-40df-bc40-ca817f01339a, groupAssignment=) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:45,107] DEBUG SyncGroup for group MirrorMaker_hkg1 failed due to coordinator rebalance (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:45,107] INFO (Re-)joining group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:45,122] DEBUG Sending JoinGroup ((type: JoinGroupRequest, groupId=MirrorMaker_hkg1, sessionTimeout=10000, rebalanceTimeout=300000, memberId=MirrorMaker_hkg1-0-3f369f11-3ce6-40df-bc40-ca817f01339a, protocolType=consumer, groupProtocols=org.apache.kafka.common.requests.JoinGroupRequest$ProtocolMetadata@10a74ad1)) to coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:45,123] DEBUG SyncGroup for group MirrorMaker_hkg1 failed due to coordinator rebalance (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:45,123] INFO (Re-)joining group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:45,138] DEBUG Sending JoinGroup ((type: JoinGroupRequest, groupId=MirrorMaker_hkg1, sessionTimeout=10000, rebalanceTimeout=300000, memberId=MirrorMaker_hkg1-1-9e6708fc-1eb1-4a62-89d5-387d78865e9e, protocolType=consumer, groupProtocols=org.apache.kafka.common.requests.JoinGroupRequest$ProtocolMetadata@5720e7e1)) to coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:48,521] DEBUG Received successful JoinGroup response for group MirrorMaker_hkg1: {error_code=0,generation_id=3,group_protocol=roundrobin,leader_id=MirrorMaker_hkg1-0-8c7d481e-8d2d-403c-acb0-c43486a1da59,member_id=MirrorMaker_hkg1-0-3f369f11-3ce6-40df-bc40-ca817f01339a,members=[]} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:48,521] DEBUG Sending follower SyncGroup for group MirrorMaker_hkg1 to coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null): (type=SyncGroupRequest, groupId=MirrorMaker_hkg1, generationId=3, memberId=MirrorMaker_hkg1-0-3f369f11-3ce6-40df-bc40-ca817f01339a, groupAssignment=) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:48,524] DEBUG Received successful JoinGroup response for group MirrorMaker_hkg1: {error_code=0,generation_id=3,group_protocol=roundrobin,leader_id=MirrorMaker_hkg1-0-8c7d481e-8d2d-403c-acb0-c43486a1da59,member_id=MirrorMaker_hkg1-1-9e6708fc-1eb1-4a62-89d5-387d78865e9e,members=[]} (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:48,524] DEBUG Sending follower SyncGroup for group MirrorMaker_hkg1 to coordinator app458.sjc2.mytest.com.com:9092 (id: 2147483613 rack: null): (type=SyncGroupRequest, groupId=MirrorMaker_hkg1, generationId=3, memberId=MirrorMaker_hkg1-1-9e6708fc-1eb1-4a62-89d5-387d78865e9e, groupAssignment=) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:51,968] FATAL [mirrormaker-thread-0] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)

org.apache.kafka.common.KafkaException: Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:548)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:521)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:784)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:765)

	at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:186)

	at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:149)

	at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:116)

	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:493)

	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:322)

	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:253)

	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:172)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:347)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)

	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)

	at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1029)

	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)

	at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:625)

	at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:431)

[2017-06-08 15:25:51,969] FATAL [mirrormaker-thread-1] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)

org.apache.kafka.common.KafkaException: Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:548)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:521)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:784)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:765)

	at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:186)

	at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:149)

	at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:116)

	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:493)

	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:322)

	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:253)

	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:172)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:347)

	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)

	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)

	at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1029)

	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)

	at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:625)

	at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:431)

[2017-06-08 15:25:51,977] INFO [mirrormaker-thread-1] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,977] INFO [mirrormaker-thread-0] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,978] INFO [mirrormaker-thread-1] Committing consumer offsets. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,979] INFO Exiting on send failure, skip committing offsets. (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:51,979] INFO [mirrormaker-thread-1] Shutting down consumer connectors. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,979] INFO [mirrormaker-thread-0] Committing consumer offsets. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,979] INFO Exiting on send failure, skip committing offsets. (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:51,980] INFO [mirrormaker-thread-0] Shutting down consumer connectors. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,981] DEBUG Sending synchronous auto-commit of offsets {} for group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2017-06-08 15:25:51,981] DEBUG Sending synchronous auto-commit of offsets {} for group MirrorMaker_hkg1 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2017-06-08 15:25:51,981] DEBUG Heartbeat thread for group MirrorMaker_hkg1 has closed (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:51,981] DEBUG Heartbeat thread for group MirrorMaker_hkg1 has closed (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[2017-06-08 15:25:51,983] DEBUG Removed sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,983] DEBUG Removed sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,983] DEBUG Removed sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,983] DEBUG Removed sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,983] DEBUG Removed sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,983] DEBUG Removed sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,984] DEBUG Removed sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,984] DEBUG Removed sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,985] DEBUG Removed sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,985] DEBUG Removed sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,986] DEBUG Removed sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,986] DEBUG Removed sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,986] DEBUG Removed sensor with name node--5.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,987] DEBUG Removed sensor with name node--5.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,987] DEBUG Removed sensor with name node--5.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,987] DEBUG Removed sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,988] DEBUG Removed sensor with name node--4.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,988] DEBUG Removed sensor with name node--4.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,988] DEBUG Removed sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,989] DEBUG Removed sensor with name node--4.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,991] DEBUG Removed sensor with name node-2147483613.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,992] DEBUG Removed sensor with name node-2147483613.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,992] DEBUG Removed sensor with name node--2.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,993] DEBUG Removed sensor with name node-2147483613.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,994] DEBUG The Kafka consumer has closed. (org.apache.kafka.clients.consumer.KafkaConsumer)

[2017-06-08 15:25:51,995] DEBUG Removed sensor with name node--2.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,995] DEBUG Removed sensor with name node--2.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,995] INFO [mirrormaker-thread-1] Mirror maker thread stopped (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,995] DEBUG Removed sensor with name node--5.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,996] DEBUG Removed sensor with name node--5.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,996] DEBUG Removed sensor with name node--5.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,996] FATAL [mirrormaker-thread-1] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,997] DEBUG Removed sensor with name node--4.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,997] DEBUG Removed sensor with name node--4.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,997] DEBUG Removed sensor with name node--4.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,998] DEBUG Removed sensor with name node-2147483613.bytes-sent (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,998] DEBUG Removed sensor with name node-2147483613.bytes-received (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,998] INFO Start clean shutdown. (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:51,998] DEBUG Removed sensor with name node-2147483613.latency (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:51,998] DEBUG The Kafka consumer has closed. (org.apache.kafka.clients.consumer.KafkaConsumer)

[2017-06-08 15:25:51,998] INFO [mirrormaker-thread-0] Mirror maker thread stopped (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,999] INFO Shutting down consumer threads. (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:51,999] INFO [mirrormaker-thread-0] Mirror maker thread shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:51,999] INFO [mirrormaker-thread-1] Mirror maker thread shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:52,000] INFO [mirrormaker-thread-0] mirrormaker-thread-0 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:52,000] INFO [mirrormaker-thread-1] mirrormaker-thread-1 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:52,001] INFO [mirrormaker-thread-0] Mirror maker thread shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:52,001] INFO [mirrormaker-thread-1] Mirror maker thread shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)

[2017-06-08 15:25:52,001] INFO Closing producer. (kafka.tools.MirrorMaker$)

[2017-06-08 15:25:52,001] INFO Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

[2017-06-08 15:25:52,002] DEBUG Beginning shutdown of Kafka producer I/O thread, sending remaining records. (org.apache.kafka.clients.producer.internals.Sender)

[2017-06-08 15:25:52,002] DEBUG Removed sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:52,002] DEBUG Removed sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:52,003] DEBUG Removed sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:52,003] DEBUG Removed sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:52,004] DEBUG Removed sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:52,004] DEBUG Removed sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:52,004] DEBUG Removed sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)

[2017-06-08 15:25:52,004] DEBUG Shutdown of Kafka producer I/O thread has completed. (org.apache.kafka.clients.producer.internals.Sender)

[2017-06-08 15:25:52,005] DEBUG The Kafka producer has closed. (org.apache.kafka.clients.producer.KafkaProducer)

[2017-06-08 15:25:52,005] INFO Kafka mirror maker shutdown successfully (kafka.tools.MirrorMaker$)

Can anyone please help me or point me to what could be the reason to this to shutdown.

Also how you set the producer to be async, it seem that the producer.type option doesnt work anymore

Thank you

Don't have an account?
Coming from Hortonworks? Activate your account here