Java application connecting to CDP DataHub Kafka

[#1]

I often encounter the following question - "How to connect a Java application to CDP Kafka?". The good news is, it's simple.

Public docs: Connecting Kafka clients to Data Hub provisioned clusters

CDP Kafka, by default, is secure and governed. That means CDP Kafka uses TLS and CDP workload users for security and authorization. CDP Kafka supports all Apache Kafka authentication capabilities. Again, by default, CDP Kafka is automatically secured <SASL_SSL> and governed by Apache Ranger.  

Prerequisites

CDP Public Cloud with a Kafka DataHub instance.

Connectivity requirements

To connect to CDP Kafka, the following is required:

• CDP TrustStore
• CDP Workload Username/Password
• Kafka Broker Name
• JaaS File

Java Example

 Properties properties = new Properties();
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "<KAFKA-BROKER>:9093,<KAFKA-BROKER>:9093,<KAFKA-BROKER>:9093");
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        properties.put("security.protocol", "SASL_SSL");
        properties.put("sasl.mechanism", "PLAIN");
        properties.put(ProducerConfig.CLIENT_ID_CONFIG, "connectivity-testing");
        properties.put("sasl.kerberos.service.name", "kafka");


        properties.put("ssl.truststore.location", "/home/sunilemanjee/truststore.jks");
        properties.put("sasl.jaas.config",
                "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"<CDP-WORKLOAD-USER>\" password=\"<CDP-WORKLOAD-PASSWORD>\";");

Full code here.

TrustStore 

Fetch the TrustStore for CDP is well documented here. 

CDP Workload Username/Password

Set and fetch CDP workload username/password

Kafka Broker Names

Fetch broker names from the CDP management console under DataHub, Hardware Tab

JaaS File

KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="cdp-workload-user"
password="cdp-workload-password;
};

That's it. Enjoy