Support Questions
Find answers, ask questions, and share your expertise

HDP 3.1.0 - Kafka 2.0.0 - Bypassing SSL Endpoint Identification

HDP 3.1.0 - Kafka 2.0.0 - Bypassing SSL Endpoint Identification

Explorer

I have an HDP cluster that I recently upgraded from 2.6.5 to 3.1.0. For Kafka, I only have the SSL listeners enabled but I've had issue with getting the certs right so in my calling apps (producer and consumer) I'm bypassing the SSL Endpoint Identification. With the change to Kafka 2.0.0 my calling apps seem to be fine, however when I try to spin up a console-consumer/producer I get the following error:

ERROR Authentication failed: terminating consumer process (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
        at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
        at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:125)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:230)
        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:314)
        at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1181)
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
        at kafka.tools.ConsoleConsumer$ConsumerWrapper.<init>(ConsoleConsumer.scala:387)
        at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:71)
        at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:53)
        at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
        ... 17 more
Caused by: java.security.cert.CertificateException: No name matching {DNS NAME OF VM REMOVED} found
        at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:221)
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:95)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
        ... 26 more

In my server.properties both ssl.endpoint.identification.algorithm and listener.name.internal.ssl.endpoint.identification.algorithm are set to empty strings (I've tried null too).

Here is how I'm calling the producer and consumer:

/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-consumer.sh --bootstrap-server myvm.mydomain.com:9093 --topic test_topic --consumer.config /usr/hdp/3.1.0.0-78/kafka/config/client-ssl.properties

/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list myvm.mydomain:9093 --topic test_topic --producer.config /usr/hdp/3.1.0.0-78/kafka/config/client-ssl.properties

Any tips on how I can get these working?

1 REPLY 1
Highlighted

Re: HDP 3.1.0 - Kafka 2.0.0 - Bypassing SSL Endpoint Identification

Explorer

Figured it out... You have to specify a null SSL Endpoint Identification Algorithm when you call the console producer/consumer.

/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-consumer.sh --bootstrap-server myvm.mydomain.com:9093 --topic test_topic --consumer.config /usr/hdp/3.1.0.0-78/kafka/config/client-ssl.properties --consumer-property ssl.endpoint.identification.algorithm=

/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list myvm.mydomain:9093 --topic test_topic --producer.config /usr/hdp/3.1.0.0-78/kafka/config/client-ssl.properties --producer-property ssl.endpoint.identification.algorithm=