Created 01-29-2025 04:42 AM
Hello,
I have got a JXM exporter configured for Kafka brokers to send metrics to prometheus which allows me to monitor Kafka via Grafana Dashboards. All is working okay, however, stuck with a weird issue.
Backaground:
JMX exporter is configured using below property for Kafka brokers:
in Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) have added a key KAFKA_JMX_OPTS and its value as -javaagent:/var/lib/prometheus_jmx_config/jmx_prometheus_javaagent-0.20.0.jar=9091:/var/lib/prometheus_jmx_config/kafka_jmx_exporter.yml.
The jar and yml files provided in the configuration are in place and all works okay. We are able to get required metrics to Grafana via Prometheus.
Issue:
Whenever we attempt to restart Kafka brokers, it does happily, however, when attempted a rolling restart it fails with below error in stderr log file:
+ [[ healthy partitions stay healthy == *\p\a\r\t\i\t\i\o\n\s\ \s\t\a\y\ \h\e\a\l\t\h\y* ]]
+ call_kafka_topics --at-min-isr-partitions
+ option=--at-min-isr-partitions
+ get_property bootstrap.servers /var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/rolling_restart_check_before_stop_admin_client_configs.properties BOOTSTRAP_SERVERS
++ dirname /opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ GET_PROPERTY_PY_DIR=/opt/cloudera/cm-agent/service/common
++ python -u /opt/cloudera/cm-agent/service/common/get_property.py bootstrap.servers /var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/rolling_restart_check_before_stop_admin_client_configs.properties
+ value=
+ eval 'BOOTSTRAP_SERVERS='\'''\'''
++ BOOTSTRAP_SERVERS=
+ [[ -z '' ]]
+ BOOTSTRAP_SERVERS=<broker>:6667
+ kafka-topics --at-min-isr-partitions --describe --bootstrap-server <broker>:6667 --command-config /var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/rolling_restart_check_before_stop_admin_client_configs.properties
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:459)
at java.base/sun.nio.ch.Net.bind(Net.java:448)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
at jdk.httpserver/sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:142)
at jdk.httpserver/sun.net.httpserver.HttpServerImpl.<init>(HttpServerImpl.java:50)
at jdk.httpserver/sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35)
at jdk.httpserver/com.sun.net.httpserver.HttpServer.create(HttpServer.java:137)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$Builder.build(HTTPServer.java:365)
at io.prometheus.jmx.common.http.HTTPServerFactory.createHTTPServer(HTTPServerFactory.java:123)
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:60)
... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
/var/run/cloudera-scm-agent/process/18315-kafka-KAFKA_BROKER-kafka_broker_rolling_restart_pre_check/scripts/broker_rolling_restart_checker.sh: line 63: 24738 Aborted kafka-topics ${option} --describe --bootstrap-server "${BOOTSTRAP_SERVERS}" --command-config "${prop_file}" >> "${describe_partitions_output}"
As per my understanding, the error indicates that a port is already being used while doing the health checks of Kafka topics. I assume it is complaining about 6667 since there is not clear mention of which port is an issue. However, rolling restart without this configuration just works fine. So confused if it is really the failure of Kafka-topics health check?
Diagnosis / troubleshooting / findings:
Unsure on what exactly have been configured incorrectly or what is missed. Request to please share your thoughts / recommendations / suggestions on the same.
Thanks
snm1523
Created 02-03-2025 01:48 AM
For the benefit of anyone if looking for a solution to this. We need to add the -javaagent parameter in "Additional Broker Java Options" configuration property instead of what mentioned in in the post.