Kafka Connect service for CDH 6.3

[#1]

Hello,   We are currently using CDH 6.3 with Kafka as one of the components.  We have Kafka and Zookeeper services installed and running.   Want to install the Kafka Connect service to install and run a few source and sink connectors.  Is it possible to do so with this distribution.

[#2]

I read in the documentation that this can be enabled using the SMM (Streams Messaging Manager) service.  It can be added using the Add a Service option next to the cluster name.  I don't see this as one of the installable options in the list of components

[#3 · Accepted solution]

@kkhambadkone1 , You are unable to find it under add service because Kafka Connect is included in CDH 6.3.x, but is not supported. Flume and Sqoop are proven solutions for batch and real time data loading that complement Kafka's message broker capability.

In order to use SMM, you need to download parcels from the official cloudera download portal but in order to do that you must be a CSM customer to access these downloads.

Having said that, please find the steps below :

#Assumption : You already have CDH 6.x and Kerberos enabled

1. Install a database

In this case, we are using MySQL:

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-installing_mysql.html

2. Configure the database for schema registry and SMM

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-configuring-schema-registry-metadata-store...

3. Download Schema Registry and SMM parcels

• SMM https://www.cloudera.com/downloads/cdf/csm.html
• CSR https://www.cloudera.com/downloads/cdf/csp.html

4. Install the Parcels

Install the services in this order:

1. Schema Registry

2. SRM (if no SRM installation, avoid this step)

3. SMM

https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-get-parcel-csd.html

5. Distribute and activate the parcels.

• In Schema registry point “Schema Registry storage connector url” to the mysql hostname. Check “Enable Kerberos Authentication”.
• Use the database registry password for “Schema Registry storage connector password”

  5.1 For SMM use

  • cm.metrics.host = cloudera manager host
  • cm.metrics.password = cloudera manager UI password
  • cm.metrics.service.name = kafka (default)
  • Streams Messaging Manager storage connector url = jdbc:mysql://FQDN_MYHSQL:3306/streamsmsgmgr
  • Streams Messaging Manager storage connector password = user database password specified
  • Check “Enable Kerberos Authentication”

6. Add Kafka service

• Check "Enable Kerberos Authentication"

7. Configure and access the SMM UI

• Property "cm.metrics.service.name" must match with the Kafka service name, by default is "kafka"
• Create streamsmsgmgr principal in the KDC, example when using MIT KDC
  • kadmin.local
  • add_principal streammsmmgr
• Finally copy the /etc/krb5.conf to your local machine and get a valid kerberos ticket for streammsmmgr user by using "kinit streammsmmgr" and use the same password chosen for the user creation time.
  
  Please hit "accept as solution" if your queries have been answered 

Regards,

Aditya

[#4]

@kkhambadkone1 Has your issue been resolved? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.