Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka console-consumer/-producer not working after adding SASL_PLAINTEXT

Kafka console-consumer/-producer not working after adding SASL_PLAINTEXT

Guru

Hi,

after enabling SASL_PLAINTEXT listener on kafka it is no longer possible to use console-consumer/-producer. Whereas using a simple Java snippet to create a producer and adding some messages, it works fine, by using the exact same user/password as used for the console-clients:

public class SimpleProducer {
    public static void main(String[] args) throws Exception{
        if(args.length == 0){
            System.out.println("Enter topic name");
            return;
        }
        String topicName = args[0].toString();
        Properties props = new Properties();
        props.put("bootstrap.servers", "<brokernode>:6666");
        props.put("acks", "1");
        props.put("retries", 0);
        props.put("batch.size", 16384);
        props.put("linger.ms", 1);
        props.put("buffer.memory", 33554432);
        props.put("key.serializer",
            "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer",
            "org.apache.kafka.common.serialization.StringSerializer");
        ////// AUTHENTICATION
        props.put("security.protocol","SASL_PLAINTEXT");
        props.put("sasl.mechanism","PLAIN");
        props.put("sasl.jaas.config",
            "org.apache.kafka.common.security.plain.PlainLoginModule required\n" +
            "username=\"kafka\"\n" +
            "password=\"kafkaSecure\";");

        ////// END AUTHENTICATION
        Producer<String, String> producer = new KafkaProducer<String, String>(props);
        System.out.println("producer created");
        for(int i = 0; i < 10; i++) {
            System.out.println("message"+i);
            producer.send(new ProducerRecord<String, String>(topicName,
                Integer.toString(i), Integer.toString(i)));
        }
        System.out.println("Messages sent successfully");
        producer.close();
    }
}

After starting the e.g. producer and trying to add a message via the console, the following message is shown (endless):

[2017-11-14 16:48:23,039] WARN Bootstrap broker <brokernode>:6666 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-11-14 16:48:23,091] WARN Bootstrap broker <brokernode>:6666 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-11-14 16:48:23,143] WARN Bootstrap broker <brokernode>:6666 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-11-14 16:48:23,195] WARN Bootstrap broker <brokernode>:6666 disconnected (org.apache.kafka.clients.NetworkClient)

Kafka config looks like:

listeners=PLAINTEXT://<brokernode>:6667,SASL_PLAINTEXT://<brokernode>:6666
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT

The console-producer gets started via:

export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/conf/user_kafka_jaas.conf" ; /usr/hdf/current/kafka-broker/bin/kafka-console-producer.sh --broker-list <brokernode>:6666 --topic gk-test --producer.config /etc/kafka/conf/producer.properties

where the property files look like:

/etc/kafka/conf/user_kafka_jaas.conf

KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="kafka"
  password="kafkaSecure";
};

/etc/kafka/conf/producer.properties

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

Any hint on what is going wrong with console-producer and console-consumer to not being able to produce/consume from topic ? ...but the Java snippet works...

Thanks

2 REPLIES 2
Highlighted

Re: Kafka console-consumer/-producer not working after adding SASL_PLAINTEXT

Explorer

I think your problem is similar to this one

https://community.hortonworks.com/questions/120978/hdp-kafka-service-setup-with-saslplain-not-workin...

I did what @Jeff Groves said, it worked. You might have a try.

Highlighted

Re: Kafka console-consumer/-producer not working after adding SASL_PLAINTEXT

@Gerd Koenig

Is your broker using the jaas (username/password) you created?

The SASL/PLAIN configuration is well documented here, Do ensure the username and password are in the jaas which broker loading in classpath.

Don't have an account?
Coming from Hortonworks? Activate your account here