Created on 01-07-202101:20 PM - edited on 01-10-202107:42 PM by subratadas
In this article, we will walk through the steps required to connect a Spark Structured Streaming application to Kafka in CDP Data Hub. We use two Data Hubs, one with a Data Engineering Template, and another with a Streams Messaging template. Both Data Hubs were created in the same environment.
1. Obtain the FreeIPA certificate of your environment:
From the CDP Home Page, navigate to Management Console > Environments
Locate and select your environment from the list of available environments
Select Get FreeIPA Certificate from the drop-down menu. The FreeIPA certificate downloads.
2. Add the FreeIPA certificate to the truststore of the client.
The certificate needs to be added for all clients that you want to connect to the Data Hub provisioned cluster. The exact steps of adding the certificate to the truststore depends on the platform and key management software used. For example, you can use the Java keytool command line tool:
Note, in the above code, we have specified our keystore location in our option (kafka.ssl.truststore.location), and our keystore password in the kafka.ssl.truststore.password option. The password we provide here is the password that we provided for our keystore at the time of its creation.
Note: We have specified our workload username and password in the "kafka.sasl.jaas.config" option.
5. Kinit as a user with permissions to the Kafka topic