Support Questions

Find answers, ask questions, and share your expertise

Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

avatar
Expert Contributor

client-sslproperties.txt

Hello - i've enabled SSL for Kafka, and Kafka is starting up fine with SSL enable.

However, when i run the Kafka console producer, it is give me error as shown below ->

Command :

/usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list nwk2-bdp-kafka-05.gdcs-qa.apple.com:6667,nwk2-bdp-kafka-04.gdcs-qa.apple.com:6667,nwk2-bdp-kafka-06.gdcs-qa.apple.com:6667 --topic sslTopic --producer.config /tmp/ssl-kafka/client-ssl.properties

Message Typed on console :

hi

On Typing message on the Console Producer, i get the following error :

[2017-07-24 19:10:22,940] WARN Bootstrap broker nwk2-bdp-kafka-06.gdcs-qa.apple.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-07-24 19:10:23,106] WARN Bootstrap broker nwk2-bdp-kafka-05.gdcs-qa.apple.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-07-24 19:10:23,160] WARN Bootstrap broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-07-24 19:10:23,208] WARN Bootstrap broker nwk2-bdp-kafka-05.gdcs-qa.apple.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-07-24 19:10:23,260] WARN Bootstrap broker nwk2-bdp-kafka-06.gdcs-qa.apple.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)

Attached is the client-ssl.properties file, used to start the Console produce

15 REPLIES 15

avatar

@Karan Alang

1) After enabling the debug - what can you see in controller log file?

2) What steps did you follow to enable SSL for Kafka?

avatar
Expert Contributor

@Daniel Kozlowski - here is the update on the this.

with -> security.inter.broker.protocol = PLAINTEXT, i'm able to start the Console Producer & consumer and publish & read the messages published.

However, when i set -> security.inter.broker.protocol = SSL, the errors start. (in both PLAINTEXT & SSL modes)

when i start the Console Producer in PLAINTEXT Mode :

/usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list nwk2-bdp-kafka-04.gdcs-qa.ale.com:6667 --topic sslTopic3 --security-protocol PLAINTEXT
[2017-07-26 05:53:26,172] WARN Error while fetching metadata with correlation id 17 : {sslTopic3=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-07-26 05:53:26,277] WARN Error while fetching metadata with correlation id 18 : {sslTopic3=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-07-26 05:53:26,388] WARN Error while fetching metadata with correlation id 19 : {sslTopic3=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

when i start the Console Producer in SSL Mode :

/usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list nwk2-bdp-kafka-04.gdcs-qa.ale.com:6668 --topic sslTopic3 --producer.config /tmp/ssl-kafka/client-ssl.properties --security-protocol SSL
hi
HELLO
[2017-07-26 05:59:31,888] ERROR Error when sending message to topic sslTopic3 with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

Error in controller.log file :

[2017-07-26 05:58:49,535] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 05:58:49,853] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

when i describe the topic, i see that the leader is 1001 & Isr has only 1001

/usr/hdp/2.5.3.0-37/kafka/bin/kafka-topics.sh --describe --zookeeper nwk2-bdp-kafka-05.gdcs-qa.apple.com:2181,nwk2-bdp-kafka-04.gdcs-qa.apple.com:2181,nwk2-bdp-kafka-06.gdcs-qa.apple.com:2181 --topic sslTopic3
Topic:sslTopic3PartitionCount:3ReplicationFactor:3Configs:
Topic: sslTopic3 Partition: 0 Leader: 1001 Replicas: 1003,1001,1002 Isr: 1001
Topic: sslTopic3 Partition: 1 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1001
Topic: sslTopic3 Partition: 2 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001

It seems setting the parameter -> security.inter.broker.protocol = SSL causes connectivity issues between the Controller (in this case 1001) & the Brokers (1001, 1002, 1003)

The question is why & what needs to be done to fix this ?

avatar
Expert Contributor

@Daniel Kozlowski - when i telnet (controller to broker, i.e 1001 -> 1001), here is what i seem..

seems connectivity on the ssl port is Not an issue

[root@nwk2-bdp-kafka-04 kafka]# telnet nwk2-bdp-kafka-04.gdcs-qa.apple.com 6668
Trying 10.78.130.143...
Connected to nwk2-bdp-kafka-04.gdcs-qa.apple.com.
Escape character is '^]'.

avatar
Expert Contributor

@Daniel Kozlowski -

here are the steps i did to enable SSL :

1) create SSL certificates, create CA using openSSL, sign the certificates using the CA & import the certificate and the signed certificate into the keystore (Steps shown below)

1) GENERATE THE JKS FILE
(Generate the key and certificate for a component process)
keytool -genkey -keyalg RSA -alias nwk2-bdp-kafka-04.gdcs-qa.apple.com -keystore server.keystore.jks -storepass changeit -validity 360 -keysize 2048


2) Get the certificate signed (using OpenSSL) - DONE
CREATE THE CERTIFICATE AUTHORITY (CA)
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

ADD THE GENERATED CA TO THE SERVER'S TRUSTSTORE: 
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert

ADD THE GENERATED CA TO THE CLIENT'S TRUSTSTORE, SO THAT CLIENTS KNOW THAT THEY CAN TRUST THIS CA
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert

3) Sign all certificates generated in Step 1 with the CA generated in Step 2a:

-> EXPORT THE CERTIFICATE FROM THE KEYSTORE
keytool -keystore server.keystore.jks -alias nwk2-bdp-kafka-04.gdcs-qa.apple.com -certreq -file cert-file

-> SIGN THE CERTIFICATE WITH THE CA

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 360 -CAcreateserial -passin pass:changeit

-> IMPORT THE CA CERTIFICATE AND THE SIGNED CERTIFICATE INTO THE KEYSTORE.

keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed

2) created client.keystore.jks

STEPS

1) GENERATE THE JKS FILE :
keytool -genkey -keyalg RSA -alias nwk2-bdp-kafka-04.gdcs-qa.apple.com -keystore client.keystore.jks -storepass changeit -validity 360 -keysize 2048


2) EXPORT THE CERTIFICATE FROM THE KEYSTORE :
keytool -keystore client.keystore.jks -alias nwk2-bdp-kafka-04.gdcs-qa.apple.com -certreq -file cert1-file


3) SIGN THE CERTIFICATE WITH THE CA

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert1-file -out cert1-signed -days 360 -CAcreateserial -passin pass:changeit

4) IMPORT THE CA CERTIFICATE AND THE SIGNED CERTIFICATE INTO THE KEYSTORE.
keytool -keystore client.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore client.keystore.jks -alias localhost -import -file cert1-signed

3) Made changes to server.properties (shared earlier)

Restarted Zookeeper & Kafka.

wrt changing the log4j.rootLogger parameter in /etc/kafka/conf/tools-log4j.properties, i'd changed the mode to DEBUG, but it seems to be getting reverted back to "WARN" when i restart the Broker .. How do i ensure it doesn't get reverted back?

Currently, the error message in the controller.log is same as shared in earlier post.

avatar
Expert Contributor

@Daniel Kozlowski - re-attaching snippet of the controller.log file

at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 07:09:00,073] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 07:09:00,406] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 07:09:00,725] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-07-26 07:09:01,045] WARN [Controller-1001-to-broker-1001-send-thread], Controller 1001's connection to broker nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to nwk2-bdp-kafka-04.gdcs-qa.apple.com:6668 (id: 1001 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) 

avatar

@Karan Alang

Re-implement the SSL by following up exactly the steps described in here:

http://docs.confluent.io/2.0.0/kafka/ssl.html