Error in Kafka startup with the JMX exporter - java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds

[#1]

Hi All,

I'm trying to integrate Prometheus to collect Kafka metrics.

I'm getting error in starting up Kafka, with the JMX exporter passed as java agent. Error is as shown below, any ideas ?

---------------------------------------------------

[root@nwk2-bdp-kafka-06 kafka]# KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/hdp/2.5.3.0-37/prometheus/jmx_prometheus_javaagent-0.9.jar=7071:/usr/hdp/2.5.3.0-37/prometheus/jmx_exporter.yaml" ./bin/kafka-server-start.sh config/server.properties 2017-07-07 00:44:31.781:INFO:ipjsoejs.Server:jetty-8.y.z-SNAPSHOT 2017-07-07 00:44:31.866:INFO:ipjsoejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:7071 Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54) at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:128) at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:121) at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:38)

[#2]

@mqureshi, @liran nahum - loopig you in, any ideas on this ?

[#3]

@Karan Alang

I have not done this but it seems like that collector is already there, at least by that name. Can you change the name and try it?

[#4 · Accepted solution]

this was fixed, there possibly seemed duplicate processes started, i killed all the services & restarted.

That fixed the issue.

[#5]

@Karan Alang

It's probably because -javaagent config was loaded twice or more in env loading.

It happens when a variable is set like this in env config : KAFKA_OPT="$KAFKA_OPTS -java-agent:..." or JAVA_OPT="$JAVA_OPTS -java-agent:..."

And variable is loaded twice during startup.