Cloudera kafka intergration with Promethous

[#1]

Hello,

I am trying to integrate Prometheus with cloudera (5.16.0) kafka cluster (kafka version = 1.0.1), but unable to export matrices.

Steps performed:

Promethous Server:

tar -xzf prometheus-*.tar.gz
mv prometheus-2.3.2.linux-amd64 prometheus
rm prometheus-*.tar.gz
cd prometheus

nano prometheus.yml
global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: 'kafka'
static_configs:
- targets:
- hadoop-2:8080 # Kafka 1 - change IP for your use case
- hadoop-3:8080 # Kafka 2 - change IP for your use case
- hadoop-4:8080 # Kafka 3 - change IP for your use case
- job_name: 'zookeeper'
static_configs:
- targets:
- hadoop-2:8080 #zookeeper
- hadoop-3:8080 #zookeeper
- hadoop-4:8080 #zookeeper

./prometheus

Kafka side:

cd /home/kumar
mkdir prometheus
cd prometheus

Downloaded jar --> jmx_prometheus_javaagent-0.3.1.jar

zookeeper-yml --> zookeeper.yaml

kafka-yml --> kafka-0-8-2.yml

Now, I wanted to add kafka & zookeeper environment as mention:

1. For Kafka
Environment="KAFKA_OPTS=-javaagent:/home/home/prometheus/jmx_prometheus_javaagent-0.3.1.jar=8080:/home/home/prometheus/kafka-0-8-2.yml"

2. For Zookeeper:

Environment="EXTRA_ARGS=-javaagent:/home/kumar/prometheus/jmx_prometheus_javaagent-0.3.1.jar=8080:/home/kumar/prometheus/zookeeper.yaml"

So my question which (safety valve) will hold this environment configuration under Kafka & Zookeeper?

While I am configuring on plain Kafka cluster, I am adding all environment argument under /etc/systemd/system/kafka.service & /etc/systemd/system/zookeeper.service for Kafka & Zookeeper respectively.

Appreciate, If any one could help me on this as I am new to Cloudera.

Thank You.

[#2]

>> You can refer to the mentioned Git Repo[1] for information on Kafka exporter for Prometheus in Kafka Manager.

[1]https://github.com/danielqsj/kafka_exporter

>> I would like to share information on SMM[2], Streams Messaging Manager is an operations monitoring and management tool from Cloudera that provides end-to-end visibility in an enterprise Apache Kafka environment.

With SMM, you can gain clear insights about your Kafka clusters. You can understand the end-to-end flow of message streams from producers to topics to consumers.

[2]https://docs.cloudera.com/csp/2.0.1/smm-overview/topics/smm-overview.html