Support Questions
Find answers, ask questions, and share your expertise

Kafka Connect service for CDH 6.3

Hello,   We are currently using CDH 6.3 with Kafka as one of the components.  We have Kafka and Zookeeper services installed and running.   Want to install the Kafka Connect service to install and run a few source and sink connectors.  Is it possible to do so with this distribution.

1 ACCEPTED SOLUTION

Cloudera Employee

@kkhambadkone1 , You are unable to find it under add service because Kafka Connect is included in CDH 6.3.x, but is not supported. Flume and Sqoop are proven solutions for batch and real time data loading that complement Kafka's message broker capability.

 

In order to use SMM, you need to download parcels from the official cloudera download portal but in order to do that you must be a CSM customer to access these downloads.

 

Having said that, please find the steps below :

 

#Assumption : You already have CDH 6.x and Kerberos enabled

1. Install a database

In this case, we are using MySQL:

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-installing_mysql.html

2. Configure the database for schema registry and SMM

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-configuring-schema-registry-metadata-store...

3. Download Schema Registry and SMM parcels

4. Install the Parcels

Install the services in this order:

 

1. Schema Registry

2. SRM (if no SRM installation, avoid this step)

3. SMM

 

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-get-parcel-csd.html

5. Distribute and activate the parcels.

  • In Schema registry point “Schema Registry storage connector url” to the mysql hostname. Check “Enable Kerberos Authentication”.
  • Use the database registry password for “Schema Registry storage connector password”

    5.1 For SMM use

    • cm.metrics.host = cloudera manager host
    • cm.metrics.password = cloudera manager UI password
    • cm.metrics.service.name = kafka (default)
    • Streams Messaging Manager storage connector url = jdbc:mysql://FQDN_MYHSQL:3306/streamsmsgmgr
    • Streams Messaging Manager storage connector password = user database password specified
    • Check “Enable Kerberos Authentication”

6. Add Kafka service

  • Check "Enable Kerberos Authentication"

7. Configure and access the SMM UI

  • Property "cm.metrics.service.name" must match with the Kafka service name, by default is "kafka"
  • Create streamsmsgmgr principal in the KDC, example when using MIT KDC
    • kadmin.local
    • add_principal streammsmmgr
  • Finally copy the /etc/krb5.conf to your local machine and get a valid kerberos ticket for streammsmmgr user by using "kinit streammsmmgr" and use the same password chosen for the user creation time.

    Please hit "accept as solution" if your queries have been answered 

Regards,

Aditya

View solution in original post

3 REPLIES 3

I read in the documentation that this can be enabled using the SMM (Streams Messaging Manager) service.  It can be added using the Add a Service option next to the cluster name.  I don't see this as one of the installable options in the list of components

Cloudera Employee

@kkhambadkone1 , You are unable to find it under add service because Kafka Connect is included in CDH 6.3.x, but is not supported. Flume and Sqoop are proven solutions for batch and real time data loading that complement Kafka's message broker capability.

 

In order to use SMM, you need to download parcels from the official cloudera download portal but in order to do that you must be a CSM customer to access these downloads.

 

Having said that, please find the steps below :

 

#Assumption : You already have CDH 6.x and Kerberos enabled

1. Install a database

In this case, we are using MySQL:

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-installing_mysql.html

2. Configure the database for schema registry and SMM

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-configuring-schema-registry-metadata-store...

3. Download Schema Registry and SMM parcels

4. Install the Parcels

Install the services in this order:

 

1. Schema Registry

2. SRM (if no SRM installation, avoid this step)

3. SMM

 

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-get-parcel-csd.html

5. Distribute and activate the parcels.

  • In Schema registry point “Schema Registry storage connector url” to the mysql hostname. Check “Enable Kerberos Authentication”.
  • Use the database registry password for “Schema Registry storage connector password”

    5.1 For SMM use

    • cm.metrics.host = cloudera manager host
    • cm.metrics.password = cloudera manager UI password
    • cm.metrics.service.name = kafka (default)
    • Streams Messaging Manager storage connector url = jdbc:mysql://FQDN_MYHSQL:3306/streamsmsgmgr
    • Streams Messaging Manager storage connector password = user database password specified
    • Check “Enable Kerberos Authentication”

6. Add Kafka service

  • Check "Enable Kerberos Authentication"

7. Configure and access the SMM UI

  • Property "cm.metrics.service.name" must match with the Kafka service name, by default is "kafka"
  • Create streamsmsgmgr principal in the KDC, example when using MIT KDC
    • kadmin.local
    • add_principal streammsmmgr
  • Finally copy the /etc/krb5.conf to your local machine and get a valid kerberos ticket for streammsmmgr user by using "kinit streammsmmgr" and use the same password chosen for the user creation time.

    Please hit "accept as solution" if your queries have been answered 

Regards,

Aditya

Community Manager

@kkhambadkone1 Has your issue been resolved? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

 

Screen Shot 2019-08-06 at 1.54.47 PM.png


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.