Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

HDP Kafka service setup with SASL/PLAIN not working

Explorer

Hi all,

We are trying to enable Kafka's SASL/PLAIN in HDP-2.6.1.0 without kerberos, and we only install 1 host for the test to ensure there is no network issue happen.

Before we enable SASL/PLAIN both kafka console producer and consumer work perfectly, after enable SASL/PLAIN the broker log seems okay.

[2017-08-08 09:41:22,101] INFO [ExpirationReaper-1004], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-08 09:41:22,102] INFO [ExpirationReaper-1004], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-08 09:41:22,108] INFO [ExpirationReaper-1004], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-08 09:41:22,133] INFO [GroupCoordinator 1004]: Starting up. (kafka.coordinator.GroupCoordinator)
[2017-08-08 09:41:22,134] INFO [GroupCoordinator 1004]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2017-08-08 09:41:22,142] INFO [Group Metadata Manager on Broker 1004]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-08-08 09:41:22,155] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2017-08-08 09:41:22,194] INFO Creating /brokers/ids/1004 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-08-08 09:41:22,205] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-08-08 09:41:22,206] INFO Registered broker 1004 at path /brokers/ids/1004 with addresses: SASL_PLAINTEXT -> EndPoint(0.0.0.0,6667,SASL_PLAINTEXT) (kafka.utils.ZkUtils)
[2017-08-08 09:41:22,219] INFO [Kafka Server 1004], started (kafka.server.KafkaServer)

But when we try to produce and consume via kafka-console script we get this error

[2017-08-08 10:03:46,936] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,041] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,143] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,245] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,347] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,449] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,551] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,654] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,756] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,858] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:47,960] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:48,062] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:48,165] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-08 10:03:48,267] WARN Bootstrap broker localhost:6667 disconnected (org.apache.kafka.clients.NetworkClient)

Any help would be appreciated.

Thanks in advance

Confing:

kafka_server_jaas.conf:

KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
   username="kafka"
   password="kafka-secret"
   user_kafka="kafka-secret"
   user_test="test-secret";
};

kafka_client_jaas.conf:

KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
  username="test"
  password="test-secret";
};

Pass the kafka_server_jaas.conf location as JVM parameter to kafka-env template:

26410-jvm-parameter-to-kafka-env-template.png

Add the properties to the Custom Kafka-broker:

26411-custom-kafka-broker.png

Change listeners:

PLAINTEXT://0.0.0.0:6667 -> SASL_PLAINTEXT://0.0.0.0:6667

Both producer.properties and consumer.properties:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

Terminal 1

$ export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/kafka/kafka_client_jaas.conf"
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:6667 --topic apple3 --from-beginning --consumer.config=/tmp/kafka/consumer.properties

Terminal 2

$ export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/kafka/kafka_client_jaas.conf"
$ bin/kafka-console-producer.sh --broker-list localhost:6667 --topic apple3 --producer.config=/tmp/kafka/producer.properties
2 REPLIES 2

Hi Mark, Did you manage to solve this problem? I am facing the exact same situation.

Cloudera Employee

Hi @Mark Lee:

Have you attempted to call the comsumer and producer with the following parameter appended to the end of the command line:

--security-protocol SASL_PLAINTEXT

As an example, your producer command line would look something like this:

bin/kafka-console-producer.sh --broker-list localhost:6667 --topic apple3 --producer.config=/tmp/kafka/producer.properties --security-protocol SASL_PLAINTEXT