Configure JMX Exporter (java agent) for Kafka - CDP Private Base 7.1.9

snm1523 — Tue, 21 Apr 2026 06:21:45 GMT

Hello,

I have got a JXM exporter configured for Kafka brokers to send metrics to prometheus which allows me to monitor Kafka via Grafana Dashboards. All is working okay, however, stuck with a weird issue.

Backaground:
JMX exporter is configured using below property for Kafka brokers:
in Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) have added a key KAFKA_JMX_OPTS and its value as -javaagent:/var/lib/prometheus_jmx_config/jmx_prometheus_javaagent-0.20.0.jar=9091:/var/lib/prometheus_jmx_config/kafka_jmx_exporter.yml.

The jar and yml files provided in the configuration are in place and all works okay. We are able to get required metrics to Grafana via Prometheus.

Issue:
Whenever we attempt to restart Kafka brokers, it does happily, however, when attempted a rolling restart it fails with below error in stderr log file:

+ [[ healthy partitions stay healthy == *\p\a\r\t\i\t\i\o\n\s\ \s\t\a\y\ \h\e\a\l\t\h\y* ]] + call_kafka_topics --at-min-isr-partitions + option=--at-min-isr-partitions + get_property bootstrap.servers /var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/rolling_restart_check_before_stop_admin_client_configs.properties BOOTSTRAP_SERVERS ++ dirname /opt/cloudera/cm-agent/service/common/cloudera-config.sh + GET_PROPERTY_PY_DIR=/opt/cloudera/cm-agent/service/common ++ python -u /opt/cloudera/cm-agent/service/common/get_property.py bootstrap.servers /var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/rolling_restart_check_before_stop_admin_client_configs.properties + value= + eval 'BOOTSTRAP_SERVERS='\'''\''' ++ BOOTSTRAP_SERVERS= + [[ -z '' ]] + BOOTSTRAP_SERVERS=<broker>:6667 + kafka-topics --at-min-isr-partitions --describe --bootstrap-server <broker>:6667 --command-config /var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/rolling_restart_check_before_stop_admin_client_configs.properties Exception in thread "main" java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525) Caused by: java.net.BindException: Address already in use at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Net.java:459) at java.base/sun.nio.ch.Net.bind(Net.java:448) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80) at jdk.httpserver/sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:142) at jdk.httpserver/sun.net.httpserver.HttpServerImpl.<init>(HttpServerImpl.java:50) at jdk.httpserver/sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35) at jdk.httpserver/com.sun.net.httpserver.HttpServer.create(HttpServer.java:137) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$Builder.build(HTTPServer.java:365) at io.prometheus.jmx.common.http.HTTPServerFactory.createHTTPServer(HTTPServerFactory.java:123) at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:60) ... 6 more *** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422 /var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/scripts/broker_rolling_restart_checker.sh: line 63: 24738 Aborted kafka-topics ${option} --describe --bootstrap-server "${BOOTSTRAP_SERVERS}" --command-config "${prop_file}" >> "${describe_partitions_output}"

As per my understanding, the error indicates that a port is already being used while doing the health checks of Kafka topics. I assume it is complaining about 6667 since there is not clear mention of which port is an issue. However, rolling restart without this configuration just works fine. So confused if it is really the failure of Kafka-topics health check?

Diagnosis / troubleshooting / findings:

When we do a full restart of kafka brokers with this configuration in place, it works okay, only the rolling restart fails.
We tried below Cloudera articles, however, no luck:
https://my.cloudera.com/knowledge/How-to-configure-JMX-access-for-Kafka-in-CDP?id=390204
Since we are on a non-SSL cluster, below was the configuration tried while following the first link:
-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+DisableExplicitGC -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.host=127.0.0.1 -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/kafka-configuring/topics/kafka-config-rolling-restart-client-conf.html
Since we are on a non-SSL cluster, below was the configuration tried while following the second link for testing:
bootstrap.servers=<broker-1>:6667,<broker-2>:6667,<broker-3>:6667,<broker-4>:6667 security.protocol=SASL_PLAINTEXT ssl.client.auth=none sasl.mechanism=GSSAPI sasl.kerberos.service.name=kafka sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/kafka.keytab" principal="<KAFKA Principal>";
Keytab location is hard coded just for 1 broker for testing and ensured that the keytab file exists at the location. We tried rolling restarting this broker, however, it failed.
Without the above JMX configuration in Advanced configuration snippet, rolling restart just works fine.

Unsure on what exactly have been configured incorrectly or what is missed. Request to please share your thoughts / recommendations / suggestions on the same.

Thanks
snm1523

Re: Configure JMX Exporter (java agent) for Kafka - CDP Private Base 7.1.9

snm1523 — Mon, 03 Feb 2025 09:48:44 GMT

For the benefit of anyone if looking for a solution to this. We need to add the -javaagent parameter in "Additional Broker Java Options" configuration property instead of what mentioned in in the post.

question Re: Configure JMX Exporter (java agent) for Kafka - CDP Private Base 7.1.9 in Support Questions

Configure JMX Exporter (java agent) for Kafka - CDP Private Base 7.1.9

Re: Configure JMX Exporter (java agent) for Kafka - CDP Private Base 7.1.9