Adding a javaagent in CDH 5 or 6.2

[#1]

We are trying to pull metrics into Promtheus and

for the life of me I spent most of lastnight trying to get CDH 6.2 to work with a javaagent.  I read a previous post where someone was have a heck of a time with jmx exporter and Ambari, so I'm assuming this is also an issue in CDH?

If I goto:

HBase Service Environment Advanced Configuration Snippet (Safety Valve)

and add this:

HBASE_MASTER_OPTS=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=10110 -javaagent:/run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar=9214:/run/cloudera-scm-agent/hbasemaster.yml"
HBASE_REGIONSERVER_OPTS=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=10111 -javaagent:/run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar=9215:/run/cloudera-scm-agent/regionserver.yml"

and then restart the service

The service WILL not restart and stderr shows:

Error opening zip file or JAR manifest missing : /run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar

I've verified all files are there :
-rw-rw-rw- 1 root root 405K Sep 4 12:20 jmx_prometheus_javaagent-0.14.0.jar
-rwxr-xr-x 1 root root 283 Oct 14 02:04 namenode.jml
-rwxr-xr-x 1 root root 283 Oct 14 02:04 datanode.jml
-rwxr-xr-x 1 root root 283 Oct 14 02:04 regionserver.jml
-rwxr-xr-x 1 root root 283 Oct 14 06:17 hbasemaster.jml
-rwxr-xr-x 1 root root 283 Oct 14 06:57 yarn.jml

I've tried to modify the hbase-env.sh directly, but of course there are so many and that I can't seem to get it to work.   Has anyone got this working in CDH5 or CDH6 and if so, can point me in the right direction

[#2 · Accepted solution]

See following resource, this fixed it for me: https://community.cloudera.com/t5/Support-Questions/Customize-Hbase-JVM-settings-in-CDH-5-4-2/td-p/3...

put everything in "HBase Service Environment Advanced Configuration Snippet (Safety Valve)"

[#3]

Hello @HadoopBD 

We hope the Steps shared by @sebastienleroy via the Community Link has worked for you. Additionally, Sharing 2 Links which document the same for ensuring HBase is monitored with Prometheus. 

- Smarak

[1] https://godatadriven.com/blog/monitoring-hbase-with-prometheus/

[2] https://grafana.com/grafana/dashboards/12722