Error in Kafka when enable TLS

[#1]

when try to create topic after enable TLS 

kafka-topics --create --topic demo --bootstrap-server 192.168.1.235:9093 --partitions 3 --replication-factor 3

this command give me this error 

Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1709199819559) timed out at 9223372036854775807 after 1 attempt(s)

24/02/29 11:42:40 INFO internals.AdminMetadataManager: [AdminClient clientId=adminclient-1] Metadata update failed

org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1709199789530) timed out at 9223372036854775807 after 1 attempt(s)

Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.

24/02/29 11:42:40 ERROR admin.TopicCommand$: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1709199819559) timed out at 9223372036854775807 after 1 attempt(s)

at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)

at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)

at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)

at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)

at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:227)

at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:196)

at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:191)

at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:219)

at kafka.admin.TopicCommand$.main(TopicCommand.scala:62)

at kafka.admin.TopicCommand.main(TopicCommand.scala)

Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1709199819559) timed out at 9223372036854775807 after 1 attempt(s)

Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.

24/02/29 11:42:40 ERROR utils.KafkaThread: Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1':

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)

at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)

at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)

at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:113)

at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:448)

at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:398)

at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)

at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)

at org.apache.kafka.common.network.Selector.poll(Selector.java:485)

at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)

at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1272)

at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1203)

at java.lang.Thread.run(Thread.java:750)

also when try to add role like kafka-connect didn't  start because this error below.

Stopping due to error
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1709200849909) timed out at 1709200849951 after 1 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1709200849909) timed out at 1709200849951 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

[#2]

@Loaimohamed79  please restart cloudera-scm-agent on all the kafka instances and check if you are still see this error.

Were u able to run this command before enabling it?

[#3]

I did that but still have the same error.

Yes, it was work before enable it.

[#4]

Hi Loaimohamed79

I see you cannot create the topics after enabling the TLS and it failed with the TimeoutException.

Since you have enabled the TLS, 'listeners' property must be using SASL_SSL or SSL protocol and hence with the 'kafka-topics' command you will need to use the argument  --command-config and pass the client.properties file that includes the below properties,
---
security.protocol=SASL_SSL OR SSL
sasl.kerberos.service.name=kafka
ssl.truststore.location=<truststore-location>
---

So the command will look like,
kafka-topics --create --topic demo --bootstrap-server 192.168.1.235:9093 --partitions 3 --replication-factor 3 --command-config <client.properties-file-location>

Kindly follow these steps and let us know if it helps.