kafka broker is not starting after configuring it for encryption with SASL_SSL protocol

[#1]

[2018-09-11 00:06:58,404] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin) [2018-09-11 00:06:58,409] INFO [Principal=kafka/kafka1.example.com@EXAMPLE.COM]: TGT refresh thread started. (org.apache.kafka.common.security.kerberos.KerberosLogin) [2018-09-11 00:06:58,409] INFO [Principal=kafka/kafka1.example.com@EXAMPLE.COM]: TGT valid starting at: Tue Sep 11 00:06:58 EDT 2018 (org.apache.kafka.common.security.kerberos.KerberosLogin) [2018-09-11 00:06:58,409] INFO [Principal=kafka/kafka1.example.com@EXAMPLE.COM]: TGT expires: Tue Sep 11 10:06:58 EDT 2018 (org.apache.kafka.common.security.kerberos.KerberosLogin) [2018-09-11 00:06:58,410] INFO [Principal=kafka/kafka1.example.com@EXAMPLE.COM]: TGT refresh sleeping until: Tue Sep 11 08:23:38 EDT 2018 (org.apache.kafka.common.security.kerberos.KerberosLogin) [2018-09-11 00:06:58,411] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: SSL trust store is specified, but trust store password is not specified. at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:94) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:93) at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:63) at kafka.network.Processor.<init>(SocketServer.scala:422) at kafka.network.SocketServer.newProcessor(SocketServer.scala:155) at kafka.network.SocketServer.$anonfun$startup$2(SocketServer.scala:96) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156) at kafka.network.SocketServer.$anonfun$startup$1(SocketServer.scala:95) at kafka.network.SocketServer.$anonfun$startup$1$adapted(SocketServer.scala:90) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.network.SocketServer.startup(SocketServer.scala:90) at kafka.server.KafkaServer.startup(KafkaServer.scala:215) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39) at kafka.Kafka$.main(Kafka.scala:67) at kafka.Kafka.main(Kafka.scala) Caused by: org.apache.kafka.common.KafkaException: SSL trust store is specified, but trust store password is not specified. at org.apache.kafka.common.security.ssl.SslFactory.createTruststore(SslFactory.java:195) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:115) at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:91) ... 16 more [2018-09-11 00:06:58,416] INFO [Kafka Server 1], shutting down (kafka.server.KafkaServer) [2018-09-11 00:06:58,421] INFO [Socket Server on Broker 1], Shutting down (kafka.network.SocketServer) [2018-09-11 00:06:58,426] WARN (kafka.utils.CoreUtils$) java.lang.NullPointerException at kafka.network.SocketServer.$anonfun$shutdown$3(SocketServer.scala:129) at kafka.network.SocketServer.$anonfun$shutdown$3$adapted(SocketServer.scala:129) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:193) at kafka.network.SocketServer.shutdown(SocketServer.scala:129) at kafka.server.KafkaServer.$anonfun$shutdown$3(KafkaServer.scala:582) at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:78) at kafka.utils.Logging.swallowWarn(Logging.scala:94) at kafka.utils.Logging.swallowWarn$(Logging.scala:93) at kafka.utils.CoreUtils$.swallowWarn(CoreUtils.scala:48) at kafka.utils.Logging.swallow(Logging.scala:96) at kafka.utils.Logging.swallow$(Logging.scala:96) at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:48) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:582) at kafka.server.KafkaServer.startup(KafkaServer.scala:289) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39) at kafka.Kafka$.main(Kafka.scala:67) at kafka.Kafka.main(Kafka.scala) [2018-09-11 00:06:58,433] INFO Shutting down. (kafka.log.LogManager) [2018-09-11 00:06:58,448] INFO Shutdown complete. (kafka.log.LogManager) [2018-09-11 00:06:58,448] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) [2018-09-11 00:06:58,555] INFO Session: 0x165b374ac140029 closed (org.apache.zookeeper.ZooKeeper) [2018-09-11 00:06:58,555] INFO EventThread shut down for session: 0x165b374ac140029 (org.apache.zookeeper.ClientCnxn) [2018-09-11 00:06:58,562] INFO [Kafka Server 1], shut down completed (kafka.server.KafkaServer) [2018-09-11 00:06:58,564] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: SSL trust store is specified, but trust store password is not specified. at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:94) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:93) at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:63) at kafka.network.Processor.<init>(SocketServer.scala:422) at kafka.network.SocketServer.newProcessor(SocketServer.scala:155) at kafka.network.SocketServer.$anonfun$startup$2(SocketServer.scala:96) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156) at kafka.network.SocketServer.$anonfun$startup$1(SocketServer.scala:95) at kafka.network.SocketServer.$anonfun$startup$1$adapted(SocketServer.scala:90) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.network.SocketServer.startup(SocketServer.scala:90) at kafka.server.KafkaServer.startup(KafkaServer.scala:215) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39) at kafka.Kafka$.main(Kafka.scala:67) at kafka.Kafka.main(Kafka.scala) Caused by: org.apache.kafka.common.KafkaException: SSL trust store is specified, but trust store password is not specified. at org.apache.kafka.common.security.ssl.SslFactory.createTruststore(SslFactory.java:195) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:115) at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:91) ... 16 more

[#2 · Accepted solution]

@Ankita Ghate
We see the error cause as following:

Caused by: org.apache.kafka.common.KafkaException: SSL trust store is specified, but trust store password is not specified.

We see that you are getting the above error in your Kafka. Which indicates that while configuring truststore for Kafka you might have forgotten to add the "ssl.truststore.password" property properly. Can you please check your Kafka configs to see if you have setup the truststore properly as mentioned in Reference Doc: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/ch_wire-kafka.html

ssl.keystore.location = /var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password = test1234
ssl.key.password = test1234
ssl.truststore.location = /var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password = test1234

Values may be based on your requirement but need to make sure that it has correctly defined ssl.truststore.password

.

[#3]

@Jay Kumar SenSharma thanks for the response.

I had provided ssl.truststore.password.generator but not ssl.truststore.password. Now I have added ssl.truststore.password kafka has started but not able to produce messages, giving error as below,

[2018-09-11 01:21:52,015] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. [2018-09-11 01:22:52,020] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.