Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

confluent sink connector consumer matrix through JMX

confluent sink connector consumer matrix through JMX

New Contributor

I am using confluent HDFS sink connector and would like to know how to get consumer properties to expose through either JMS or rest api.


I checked the following two properties, however, I don't know how to expose matrix to jmx port


1. connect-standalone.properties

2. consumer.properties

3 REPLIES 3

Re: confluent sink connector consumer matrix through JMX

Super Collaborator

JMX is not exposed in a property file. It is toggled from two environment variable. Confluent and Apache Kafka installations share the same variables for this.

Source code here -

https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L166-L174

Basically, if you export the JMX_PORT to a valid port number, it will open that port for JMX monitoring of any Kafka-related script that you run.

The recommendation, however, is to use a cluster of machines running Connect Distributed, as it is a long-running process, and you can scale your metrics collection using tools like Prometheus JMX Exporter combined with a Grafana server for dashboarding.

Re: confluent sink connector consumer matrix through JMX

New Contributor

@Jordan Moore

This is what we have done. However, I am not able to see any consumer matrix , I am only able to see broker matrix. is there something I am doing wrong.



Re: confluent sink connector consumer matrix through JMX

Super Collaborator

@Gaurang Shah

If you see broker metrics, then that is where you exposed the port to. Kafka Connect is meant to be run on separate machines from the brokers, but if you are running on the same ones, then you must expose and monitor two differerent JMX ports.