Kafka Connect service for CDH 6.3

kkhambadkone1 — Fri, 16 Sep 2022 14:41:43 GMT

Hello, We are currently using CDH 6.3 with Kafka as one of the components. We have Kafka and Zookeeper services installed and running. Want to install the Kafka Connect service to install and run a few source and sink connectors. Is it possible to do so with this distribution.

Re: Kafka Connect service for CDH 6.3

kkhambadkone1 — Sat, 22 May 2021 00:01:26 GMT

I read in the documentation that this can be enabled using the SMM (Streams Messaging Manager) service. It can be added using the Add a Service option next to the cluster name. I don't see this as one of the installable options in the list of components

Re: Kafka Connect service for CDH 6.3

LeetKrew93 — Fri, 28 May 2021 10:16:53 GMT

@kkhambadkone1 , You are unable to find it under add service because Kafka Connect is included in CDH 6.3.x, but is not supported. Flume and Sqoop are proven solutions for batch and real time data loading that complement Kafka's message broker capability.

In order to use SMM, you need to download parcels from the official cloudera download portal but in order to do that you must be a CSM customer to access these downloads.

Having said that, please find the steps below :

#Assumption : You already have CDH 6.x and Kerberos enabled

1. Install a database

In this case, we are using MySQL:

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-installing_mysql.html

2. Configure the database for schema registry and SMM

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-configuring-schema-registry-metadata-store...

3. Download Schema Registry and SMM parcels

4. Install the Parcels

Install the services in this order:

1. Schema Registry

2. SRM (if no SRM installation, avoid this step)

3. SMM

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-get-parcel-csd.html

5. Distribute and activate the parcels.

In Schema registry point “Schema Registry storage connector url” to the mysql hostname. Check “Enable Kerberos Authentication”.
Use the database registry password for “Schema Registry storage connector password”
5.1 For SMM use
- cm.metrics.host = cloudera manager host
- cm.metrics.password = cloudera manager UI password
- cm.metrics.service.name = kafka (default)
- Streams Messaging Manager storage connector url = jdbc:mysql://FQDN_MYHSQL:3306/streamsmsgmgr
- Streams Messaging Manager storage connector password = user database password specified
- Check “Enable Kerberos Authentication”

6. Add Kafka service

Check "Enable Kerberos Authentication"

7. Configure and access the SMM UI

Property "cm.metrics.service.name" must match with the Kafka service name, by default is "kafka"
Create streamsmsgmgr principal in the KDC, example when using MIT KDC
- kadmin.local
- add_principal streammsmmgr
Finally copy the /etc/krb5.conf to your local machine and get a valid kerberos ticket for streammsmmgr user by using "kinit streammsmmgr" and use the same password chosen for the user creation time.

Please hit "accept as solution" if your queries have been answered

Regards,

Aditya

Re: Kafka Connect service for CDH 6.3

cjervis — Thu, 03 Jun 2021 12:31:32 GMT

@kkhambadkone1 Has your issue been resolved? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

question Kafka Connect service for CDH 6.3 in Support Questions

Kafka Connect service for CDH 6.3

Re: Kafka Connect service for CDH 6.3

Re: Kafka Connect service for CDH 6.3

1. Install a database

2. Configure the database for schema registry and SMM

3. Download Schema Registry and SMM parcels

4. Install the Parcels

5. Distribute and activate the parcels.

5.1 For SMM use

6. Add Kafka service

7. Configure and access the SMM UI

Re: Kafka Connect service for CDH 6.3