kafka commands

roshanbi — Fri, 11 Jun 2021 11:09:52 GMT

Hello Team,

when executing kafka-topics --list I am getting error

kafka-topics --list --bootstrap-server RB-HADOOP-02.mtg.local:9092,RB-HADOOP-03.mtg.local:9092,RB-HADOOP-04.mtg.local:9092

[root@rb-hadoop-02 admin]# kafka-topics --list --bootstrap-server RB-HADOOP-02.mtg.local:9092,RB-HADOOP-03.mtg.local:9092,RB-HADOOP-04.mtg.local:9092
21/06/11 15:07:40 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
21/06/11 15:07:41 INFO admin.AdminClientConfig: AdminClientConfig values:
bootstrap.servers = [RB-HADOOP-02.mtg.local:9092, RB-HADOOP-03.mtg.local:9092, RB-HADOOP-04.mtg.local:9092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 120000
retries = 5
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS

21/06/11 15:07:41 DEBUG internals.AdminMetadataManager: [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [RB-HADOOP-04.mtg.local:9092 (id: -3 rack: null), rb-hadoop-02.mtg.local:9092 (id: -1 rack: null), RB-HADOOP-03.mtg.local:9092 (id: -2 rack: null)], partitions = [], controller = null).
21/06/11 15:07:41 INFO utils.AppInfoParser: Kafka version: 2.4.1.7.1.5.2-1
21/06/11 15:07:41 INFO utils.AppInfoParser: Kafka commitId: a27f0fc2ed4667da
21/06/11 15:07:41 INFO utils.AppInfoParser: Kafka startTimeMs: 1623409661406
21/06/11 15:07:41 DEBUG admin.KafkaAdminClient: [AdminClient clientId=adminclient-1] Kafka admin client initialized
21/06/11 15:07:41 DEBUG admin.KafkaAdminClient: [AdminClient clientId=adminclient-1] Queueing Call(callName=listTopics, deadlineMs=1623409781410) with a timeout 120000 ms from now.
21/06/11 15:07:41 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Initiating connection to node RB-HADOOP-03.mtg.local:9092 (id: -2 rack: null) using address RB-HADOOP-03.mtg.local/10.10.104.107
21/06/11 15:07:41 DEBUG network.Selector: [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2
21/06/11 15:07:41 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Completed connection to node -2. Fetching API versions.
21/06/11 15:07:41 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -2.
21/06/11 15:07:41 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Recorded API versions for node -2: (Produce(0): 0 to 8 [usable: 8], Fetch(1): 0 to 11 [usable: 11], ListOffsets(2): 0 to 5 [usable: 5], Metadata(3): 0 to 9 [usable: 9], LeaderAndIsr(4): 0 to 4 [usable: 4], StopReplica(5): 0 to 2 [usable: 2], UpdateMetadata(6): 0 to 6 [usable: 6], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 6 [usable: 6], FindCoordinator(10): 0 to 3 [usable: 3], JoinGroup(11): 0 to 6 [usable: 6], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 4 [usable: 4], SyncGroup(14): 0 to 4 [usable: 4], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 3 [usable: 3], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 5 [usable: 5], DeleteTopics(20): 0 to 4 [usable: 4], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 2 [usable: 2], OffsetForLeaderEpoch(23): 0 to 3 [usable: 3], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 1], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 2 [usable: 2], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0])
21/06/11 15:07:42 DEBUG network.Selector: [AdminClient clientId=adminclient-1] Connection with RB-HADOOP-03.mtg.local/10.10.104.107 disconnected
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:547)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1196)
at java.lang.Thread.run(Thread.java:748)
21/06/11 15:07:42 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Node -2 disconnected.
21/06/11 15:07:42 DEBUG admin.KafkaAdminClient: [AdminClient clientId=adminclient-1] Call(callName=fetchMetadata, deadlineMs=1623409781409) failed: DisconnectException: Cancelled fetchMetadata request with correlation id 1 due to node -2 being disconnected. Beginning retry #1
21/06/11 15:07:42 DEBUG admin.KafkaAdminClient: [AdminClient clientId=adminclient-1] Queueing Call(callName=fetchMetadata, deadlineMs=1623409781409) with a timeout 119371 ms from now.
21/06/11 15:07:42 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Initiating connection to node RB-HADOOP-04.mtg.local:9092 (id: -3 rack: null) using address RB-HADOOP-04.mtg.local/10.10.104.108
21/06/11 15:07:42 DEBUG network.Selector: [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3
21/06/11 15:07:42 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Completed connection to node -3. Fetching API versions.
21/06/11 15:07:42 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -3.
21/06/11 15:07:42 DEBUG clients.NetworkClient: [AdminClient clientId=adminclient-1] Recorded API versions for node -3: (Produce(0): 0 to 8 [usable: 8], Fetch(1): 0 to 11 [usable: 11], ListOffsets(2): 0 to 5 [usable: 5], Metadata(3): 0 to 9 [usable: 9], LeaderAndIsr(4): 0 to 4 [usable: 4], StopReplica(5): 0 to 2 [usable: 2], UpdateMetadata(6): 0 to 6 [usable: 6], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 6 [usable: 6], FindCoordinator(10): 0 to 3 [usable: 3], JoinGroup(11): 0 to 6 [usable: 6], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 4 [usable: 4], SyncGroup(14): 0 to 4 [usable: 4], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 3 [usable: 3], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 5 [usable: 5], DeleteTopics(20): 0 to 4 [usable: 4], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 2 [usable: 2], OffsetForLeaderEpoch(23): 0 to 3 [usable: 3], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 to 1 [usable: 1], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 2 [usable: 2], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0])

Kindly advise

Thanks,

Roshan

Re: kafka commands

GangWar — Sat, 12 Jun 2021 06:59:14 GMT

@roshanbi This seems a connection issue from the broker. You may want to increase "connections.max.idle.ms" value and try.

See this : https://community.cloudera.com/t5/Support-Questions/Unable-to-push-any-messages-to-Kafka/td-p/141484

I found something related with same : https://stackoverflow.com/questions/33432027/kafka-error-in-i-o-java-io-eofexception-null

Re: kafka commands

cjervis — Wed, 23 Jun 2021 12:29:30 GMT

@roshanbi, have you resolved your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

question kafka commands in Support Questions

kafka commands

Re: kafka commands

Re: kafka commands