Skip to main content
pvidal
Silver ⭐️⭐️⭐️
pvidal
Silver ⭐️⭐️⭐️
May 8, 2020

How to connect Datahub Kafka from Cloudera Machine Learning running Jupyter python notebook

  • October 5, 2026 at 8:53 AM
  • 4 replies
  • 6685 views
  • Translate All

Here is a fun one: how do you connect from Python in Cloudera Machine Learning to our Kafka datahub cluster?

The documentation is pretty thorough, but it does not have an example of a python client. That's what I'm going to highlight in this article.

 

The good news is that since CML and Datahub run in the same network, you don't need to worry about opening the broker ports, therefore you just need to follow these steps:

  • Step 1: Get and upload your freeIPA certificate
  • Step 2: Find your broker hostnames
  • Step 3: Setup your client

Step 1: Get and upload your freeIPA certificate

  1. Go to your management console > your environment > Actions > Get FreeIPA Certificate:
    Screen Shot 2020-05-08 at 12.52.02 PM.png
  2. Once downloaded, go to your CML workspace, and upload your file (e.g. /home/cdsw/ca.crt).

Step 2: Find your broker hostnames

For this, go to your Kafka Datahub Cluster > CM UI > Kafka > Instances; you can find the broker hosts here:
Screen Shot 2020-05-08 at 1.18.53 PM.png

Step 3: Setup your client

Then, open a session in CML, and use the following parameters:

 

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers=['<YOUR_BROKER_URL>:9093','<YOUR_BROKER_URL>:9093','<YOUR_BROKER_URL>:9093'], 
security_protocol="SASL_SSL",
sasl_mechanism="PLAIN",
ssl_check_hostname=True,
ssl_cafile='/home/cdsw/ca.crt', 
sasl_plain_username="<YOUR_WORKLOAD_USER>", 
sasl_plain_password="<YOUR_WORKLOAD_PASSWORD>", 
api_version_auto_timeout_ms=30000)

 

 

    4 replies

    joyabrata
    joyabrata
    Silver ⭐️⭐️
    February 8, 2021

    @pvidal - Thanks for this quick guide.

    Following scenario for external client TLS connectivity over PAM authentication (SASL_SSL) would be much helpful if added in documentation as an end-to-end reference ;
    1. As CDP internally manages the Certificate Authority,
    2. So truststore/keystore credentials are not available with Admins unlike on-premises clusters,
    3. In that case how to import client certificates for mutual TLS setup,
    4. Also not able to see 'Get FreeIPA Certificate' though having all roles/account admin access.

    Thanks,
    Joy

    Translate
    jagadeesan
    Platinum ⭐️
    jagadeesan
    Platinum ⭐️
    February 9, 2021

    Hi @joyabrata

    Obtain the FreeIPA certificate of your environment:

    1. From the CDP Home Page navigate to Management Console > Environments.
    2. Locate and select your environment from the list of available environments.
    3. Go to the Summary tab, scroll down to FreeIPA session then click Actions
    4. Select Get FreeIPA Certificate from the drop-down menu.
    The FreeIPA certificate downloads and follows this document 
    Translate
    joyabrata
    joyabrata
    Silver ⭐️⭐️
    February 9, 2021

    @jagadeesan - Followed the same doc, please refer below screenshot.
    No_get_cert.jpg

    Translate
    jagadeesan
    Platinum ⭐️
    jagadeesan
    Platinum ⭐️
    February 9, 2021

    Hi @joyabrata 

    I think you are looking in the Data Lake tab which is a different one, you can go to the Summary tab, then scroll down to FreeIPA session then click Actions and get Get FreeIPA Certificate from the drop-down menu. Hope this will help you. 

    Translate
    Terms & ConditionsAccessibility statement