List CDP DataHub Kafka Topics using CLI

[#1]

A short how-to article on how to list CDP Data hub Kafka topics via command line:

Prerequisites

• CDP DataHub Kafka cluster
• Workload username & password
• CDP PC trust store

Steps

1. Fetch CDP cert and store in a trust store
   • Creating TLS truststore
   • or simple fetch it from any node

/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks

• ssh into any Data hub Kafka node using CDP workload username/password
• Copy trust store onto the node
• Set broker host

BROKER_HOST=$(hostname -f)

• Create jaas.conf

KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="<workload username>"
password="<workload password>";
};

• Set KAFKA_OPTS environment variable to point to location where jaas.conf is located

export KAFKA_OPTS=-Djava.security.auth.login.config=<LOCATION-OF-YOUR-JAASCONF>/jaas.conf

• Create client-kerberos.properties

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.kerberos.service.name=kafka
ssl.truststore.location=/<LOCATION-OF-YOUR-TRUSTSTORE>/truststore.jks
org.apache.kafka.common.security.plain.PlainLoginModule required username=<WORKLOAD-USER-NAME> password=<WORKLOAD-PASSWORD>

• Run Kafka-topics

/opt/cloudera/parcels/CDH/bin/kafka-topics --list --bootstrap-server $BROKER_HOST:9093 --command-config <LOCATION-OF-YOUR-CLIENT-KERB-PROP-FILE>/client-kerberos.properties

[#2]

Some more examples here: https://github.com/asdaraujo/cdp-examples