HDP Kafka service setup with SASL/PLAIN not working

[#1]

Hi all,

We are trying to enable Kafka's SASL/PLAIN in HDP-2.6.1.0 without kerberos, and we only install 1 host for the test to ensure there is no network issue happen.

Before we enable SASL/PLAIN both kafka console producer and consumer work perfectly, after enable SASL/PLAIN the broker log seems okay.

[2017-08-08 09:41:22,101] INFO [ExpirationReaper-1004], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-08 09:41:22,102] INFO [ExpirationReaper-1004], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-08 09:41:22,108] INFO [ExpirationReaper-1004], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-08 09:41:22,133] INFO [GroupCoordinator 1004]: Starting up. (kafka.coordinator.GroupCoordinator)
[2017-08-08 09:41:22,134] INFO [GroupCoordinator 1004]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2017-08-08 09:41:22,142] INFO [Group Metadata Manager on Broker 1004]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-08-08 09:41:22,155] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2017-08-08 09:41:22,194] INFO Creating /brokers/ids/1004 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-08-08 09:41:22,205] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-08-08 09:41:22,206] INFO Registered broker 1004 at path /brokers/ids/1004 with addresses: SASL_PLAINTEXT -> EndPoint(0.0.0.0,6667,SASL_PLAINTEXT) (kafka.utils.ZkUtils)
[2017-08-08 09:41:22,219] INFO [Kafka Server 1004], started (kafka.server.KafkaServer)

But when we try to produce and consume via kafka-console script we get this error

[2017-08-08 10:03:46,936] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,041] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,143] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,245] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,347] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,449] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,551] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,654] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,756] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,858] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,960] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:48,062] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:48,165] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:48,267] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)

Any help would be appreciated.

Thanks in advance

Confing:

kafka_server_jaas.conf:

KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
   username="kafka"
   password="kafka-secret"
   user_kafka="kafka-secret"
   user_test="test-secret";
};

kafka_client_jaas.conf:

KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
  username="test"
  password="test-secret";
};

Pass the kafka_server_jaas.conf location as JVM parameter to kafka-env template:

Add the properties to the Custom Kafka-broker:

Change listeners:

PLAINTEXT://0.0.0.0:6667 -> SASL_PLAINTEXT://0.0.0.0:6667

Both producer.properties and consumer.properties:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

Terminal 1

$ export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/kafka/kafka_client_jaas.conf"
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:6667 --topic apple3 --from-beginning --consumer.config=/tmp/kafka/consumer.properties

Terminal 2

$ export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/kafka/kafka_client_jaas.conf"
$ bin/kafka-console-producer.sh --broker-list localhost:6667 --topic apple3 --producer.config=/tmp/kafka/producer.properties

[#2]

Hi Mark, Did you manage to solve this problem? I am facing the exact same situation.

[#3]

Hi @Mark Lee:

Have you attempted to call the comsumer and producer with the following parameter appended to the end of the command line:

--security-protocol SASL_PLAINTEXT

As an example, your producer command line would look something like this:

bin/kafka-console-producer.sh --broker-list localhost:6667 --topic apple3 --producer.config=/tmp/kafka/producer.properties --security-protocol SASL_PLAINTEXT