Cloudera has a lot of ways to get a cluster. Almost none of them are “one command on your laptop.” cloudera-labs/cloudera-ce-aws is the exception: a Terraform + Ansible bundle that stands up a full Cloudera Private Cloud Community Edition cluster on AWS — Cloudera Manager, Kerberos, Auto-TLS, a real storage/compute topology — from a single ansible-navigator run. This post is me taking my freshly-released fork from zero to a running Ozone cluster, and the handful of real snags between the README and a green Cloudera Manager. Everything here is field-run against cloudera-ce-aws v1.0.0, deploying Cloudera Manager 7.13.2 / Runtime 7.3.2, from an Apple-Silicon Mac into AWS account AWS SE in us-east-2. What it actually deploysPermalink A ring-fenced cluster — ~11 EC2 nodes — with everything a real Cloudera deployment has and a laptop demo usually fakes: Cloudera Manager with Kerberos and Auto-TLS Self-contained DNS, Kerberos (FreeIPA), PostgreSQL, and TLS (ACME-managed certs on a Caddy reverse proxy) A selectable topology — Ozone, Kafka, Flink, NiFi, CSA, or ECS — each its own playbook Reverse HTTPS proxies + SSH as the only ways in; the cluster is otherwise sealed The whole thing is idempotent: re-running produces no unintended changes. The node roles and default sizing (t3a for most, one r5a.xlarge for CMS): Role Count Instance gateway 1 t3a.medium services 1 t3a.large masters 3 t3a.xlarge workers 4 t3a.xlarge cms 1 r5a.xlarge sdx 1 t3a.xlarge At on-demand rates that’s roughly ~$2/hr / ~$45/day — worth a pause.yml between sessions (stops EC2, keeps EBS) if you’re leaving it up. The setup is genuinely minimalPermalink Every dependency — Terraform, Ansible, all the collections — is baked into an Ansible execution-environment container image. Locally you need almost nothing: git clone https://github.com/cloudera-labs/cloudera-ce-aws.git cd cloudera-ce-aws python -m venv ~/cdp-navigator && source ~/cdp-navigator/bin/activate pip install ansible-core ansible-navigator Plus a container runtime (Docker or Podman) and two credentials: AWS SSO and a Cloudera license .txt. # AWS SSO — the config uses your SSO profile to mint short-lived creds aws sso login --profile YOUR_PROFILE # Cloudera Private Cloud license — the text file, NOT the .zip export CDP_LICENSE_FILE=/path/to/license.txt Then a three-line config.yml: name_prefix: "steven-ce" infra_region: "us-east-2" common_password: "<min 8 chars, 1 number>" owner_email: "[email protected]" And the one command that does everything: ansible-navigator run playbooks/infrastructure.yml playbooks/services.yml \ playbooks/cms.yml playbooks/ozone-cluster.yml -e @config.yml -m stdout Four playbooks, four stages: Terraform provisions the AWS infra → Ansible configures DNS/Kerberos/DB/TLS → Cloudera Manager comes up → the Ozone cluster deploys. The snags between README and a running clusterPermalink The quickstart is clean, but seven things cost me time — exactly the stuff a reveal post should call out. The first four are one-time setup friction; the last three are genuine traps in the v1.0.0 release. 1. The :latest EE image tag isn’t publishedPermalink ansible-navigator.yml points the execution environment at ghcr.io/cloudera-labs/cloudera-ce-aws:latest. That tag doesn’t exist — the registry only publishes 1.0.0-amd64: Error response from daemon: failed to resolve reference "ghcr.io/cloudera-labs/cloudera-ce-aws:latest": not found Fix — pin the real tag in ansible-navigator.yml: image: ghcr.io/cloudera-labs/cloudera-ce-aws:1.0.0-amd64 2. The EE image is amd64-only — on Apple Silicon it runs emulatedPermalink The only published arch is -amd64. On an M-series Mac the image runs under emulation; make it explicit so Docker doesn’t guess: container-options: - "--network=host" - "--platform=linux/amd64" The EE is an orchestration controller — it drives Terraform and SSHes to the nodes; it isn’t doing heavy local compute — so emulation is a non-issue for throughput here. 3. “Logged into AWS” (console) ≠ AWS CLI has credentialsPermalink I was logged into the AWS access portal in the browser, but the CLI had no profile, no cached token, nothing — NoCredentials. The fix is aws configure sso, but the trap is subtler: after setup my default profile carried the sso_session but was missing sso_account_id and sso_role_name, so it still couldn’t resolve credentials. A complete profile: [sso-session Cloudera-Main-SSO] sso_start_url = https://d-xxxxxxxxxx.awsapps.com/start#/ sso_region = us-east-1 sso_registration_scopes = sso:account:access [profile cldr-se] sso_session = Cloudera-Main-SSO sso_account_id = 007856030109 sso_role_name = cldr_poweruser region = us-east-2 aws sts get-caller-identity --profile cldr-se should return your assumed-role ARN. That token is cached on disk, so it survives across shells — which matters because the deploy consumes the creds via aws configure export-credentials. 4. common_password and config.yml are secrets in a public repoPermalink config.yml holds a plaintext password and your fork is public. It’s in .gitignore (alongside *.pem and *.tfstate) — confirm that before you commit anything, because a leaked common_password there unlocks every service in the cluster. 5. Keep common_password alphanumeric — special characters break service enrollmentPermalink  Hint — this one cost me a full teardown. Make common_password letters and digits only. Cloudera’s automation sets service admin passwords through basic-auth API calls shaped like https://admin:PASSWORD@host/..., so an @ or # inside the password corrupts the URL’s userinfo section and enrollment fails — and the task is no_log, so the error is censored and you can’t see why. Alphanumeric still satisfies the “min 8 chars, 1 number” rule; you lose nothing. My first run had common_password full of special characters (#, @). The deploy sailed through Terraform and most of the services stage, then died on: TASK [cloudera.exe.grafana : Set Grafana admin password if API login fails] fatal: [<services-node>]: FAILED! => {"censored": "... 'no_log: true' ..."} The result is censored (no_log), but the cause is the password: common_password feeds service admin passwords that get set via basic-auth API calls (https://admin:PASSWORD@host/...). An @ inside the password breaks URL userinfo parsing, so the API login/set fails. The same password later feeds CM, Ranger, Knox, Hue, and SMM — so this isn’t a Grafana quirk, it’s a landmine for every API-set credential downstream. Fix: keep common_password alphanumeric (letters + digits, meets the “min 8, 1 number” rule without @ # $ / :). Because the password is baked into FreeIPA/DB/services as they’re provisioned, the clean fix is a teardown + redeploy with the safe password, not an in-place change. 6. enable_prometheus is declared twice — Grafana runs even when you think it’s offPermalink config-template.yml implies Prometheus/Grafana is off by default (# enable_prometheus: false). But group_vars/all.yml defines the key twice — false, then true further down — and last-wins in YAML, so the effective default is true. That’s why the Grafana tasks ran (and hit gotcha #5) even though I never enabled them. If you don’t want the monitoring stack, set enable_prometheus: false explicitly in your config.yml so it overrides the duplicate. 7. tee-ing an ansible-navigator run hides the real exit codePermalink The EE launches with --tty, so piping the run through tee sends output to the container’s PTY (the pipe stays empty) and reports the pipeline’s exit code (tee’s 0) rather than ansible’s. A run that actually failed looked like it succeeded. Watch the run with docker logs -f <ansible_runner_container> instead, and trust the PLAY RECAP failed= counts, not the shell exit code. The deploy, stage by stagePermalink One ansible-navigator run chains four playbooks. On my run — Apple-Silicon Mac, amd64 EE under emulation, default instance sizes — the full stand-up took about 2.5 hours end to end. The long poles are parcel distribution and bringing 14 Kerberized services up, not the Terraform infra (which was ~10 min); native amd64 and larger nodes would cut this down. Stage Playbook What happens 1 infrastructure.yml Terraform: VPC, security groups, 11 EC2 nodes, generated SSH key 2 services.yml FreeIPA (DNS + Kerberos), PostgreSQL, Caddy/TLS, Node Exporter, Prometheus/Grafana 3 cms.yml Cloudera Manager install + license, CM agents, AutoTLS, CM Kerberos 4 ozone-cluster.yml CM builds the cluster: distribute/activate parcels, assign roles, start services Every stage ended failed=0. The final recap across all 11 hosts: PLAY RECAP steven-ce-base-master-01.cldr.internal : ok=197 changed=75 unreachable=0 failed=0 steven-ce-base-master-02.cldr.internal : ok=197 changed=75 unreachable=0 failed=0 steven-ce-base-master-03.cldr.internal : ok=197 changed=75 unreachable=0 failed=0 steven-ce-base-worker-01.cldr.internal : ok=197 changed=75 unreachable=0 failed=0 ... workers 02–04 identical ... steven-ce-gateway-01.cldr.internal : ok=93 changed=55 unreachable=0 failed=0 steven-ce-manager-01.cldr.internal : ok=182 changed=79 unreachable=0 failed=0 steven-ce-sdx-01.cldr.internal : ok=196 changed=75 unreachable=0 failed=0 steven-ce-services-01.cldr.internal : ok=205 changed=121 unreachable=0 failed=0 One thing worth knowing: right after the Ozone stage completes, the CM cluster can briefly show BAD_HEALTH while ZooKeeper’s startup canary settles — it flips to GOOD on its own within a couple minutes. Don’t panic-restart it. What you get at the endPermalink A GOOD_HEALTH ozone-base-cluster on Cloudera Runtime 7.3.2, reachable through the Caddy reverse proxy on the single public node (the gateway) via a nip.io hostname: Cloudera Manager: https://cm.<gateway-public-ip>.nip.io — admin / your common_password Cluster health (straight from the CM API): cluster GOOD_HEALTH; all 14 services GOOD — HDFS, Ozone, Kafka, YARN, Hive, Hive-on-Tez, HBase, Ranger, Knox, Atlas, Solr, ZooKeeper, Tez, Core Settings. Only the gateway node has a public IP; every other node is private and reached through the proxy — the ring-fenced design the README promises. Cost control — pause, resume, tear downPermalink The cluster bills ~$2/hr while it runs, so know the exits up front. All three are the same one-command shape: # Pause — stop the EC2 instances, keep the EBS volumes + cluster state (cheapest way to keep it around) ansible-navigator run playbooks/pause.yml -e @config.yml -m stdout # Resume — start the instances back up ansible-navigator run playbooks/resume.yml -e @config.yml -m stdout # Tear down — Terraform destroys everything: instances, volumes, VPC ansible-navigator run playbooks/infrastructure-teardown.yml -e @config.yml -m stdout Teardown is a terraform destroy under the hood and finishes in a few minutes with a clean recap: PLAY RECAP localhost : ok=3 changed=1 unreachable=0 failed=0 Then confirm nothing is left billing before you walk away — Terraform state should be empty and AWS should report zero instances: aws ec2 describe-instances --profile <your-profile> --region us-east-2 \ --filters "Name=tag:deployment,Values=<name_prefix>" \ "Name=instance-state-name,Values=running,pending,stopping,stopped" \ --query 'length(Reservations[].Instances[])' --output text # -> 0 What NOT to doPermalink Don’t trust the :latest EE tag — pin 1.0.0-amd64. Don’t assume console login = CLI creds — configure an SSO profile with account and role. Don’t commit config.yml — it holds a plaintext password; keep it gitignored. Don’t use the license .zip — CDP_LICENSE_FILE wants the .txt. Don’t leave it running unwatched — pause.yml or infrastructure-teardown.yml when you’re done. Cloudera Community Edition on AWS in One CommandPermalink If you would like a deeper dive, hands on experience, demos, or are interested in speaking with me further about Cloudera Community Edition on AWS in One Command please reach out to schedule a discussion. ... View more

If you have ever run Cloudera Edge Flow Manager (EFM) out of the box on Kubernetes, you know the sad little moment when the pod restarts and every agent class, every flow, and every resource you painstakingly uploaded is just… gone. EFM’s default persistence lives inside the pod’s ephemeral filesystem, and unless you wire it up properly it never sees a second day. This post is the recipe I keep coming back to for running EFM 2.3.1.0-2 on minikube with full persistence. All state survives kubectl rollout restart, minikube stop, or the laptop lid closing on you at the wrong moment. Three things need a permanent home: Metadata (agent classes, flows, agents, manifests) → PostgreSQL Agent binaries (MiNiFi C++ / Java installers) → a PVC Uploaded resources (Python scripts, JARs, custom assets) → a second PVC The third one is the piece a bare EFM install always loses on restart, and it is the reason this post exists. 1. PrerequisitesPermalink You will need a running minikube cluster with: Installed Cloudera Streaming Operators in the cld-streaming namespace A PostgreSQL pod (I use ssb-postgresql from the SSB stack) — this becomes EFM’s backing store Kafka pods if your flows publish to Kafka Access to container.repo.cloudera.com (your Cloudera entitlement) Confirm they are all up: kubectl get pods -n cld-streaming | grep -E "postgres|kafka" 2. Create the EFM Database in PostgreSQLPermalink EFM needs its own database and user inside the existing Postgres. One-time setup: PG=$(kubectl get pods -n cld-streaming | grep postgres | awk '{print $1}' | head -1) kubectl exec $PG -n cld-streaming -- psql -U postgres -c "CREATE DATABASE efm;" kubectl exec $PG -n cld-streaming -- psql -U postgres -c "CREATE USER efm WITH PASSWORD 'efm_password';" kubectl exec $PG -n cld-streaming -- psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE efm TO efm;" kubectl exec $PG -n cld-streaming -- psql -U postgres -c "ALTER DATABASE efm OWNER TO efm;" 3. Create the SecretsPermalink Three secrets: the DB password, the EFM encryption password, and the Cloudera registry pull secret. kubectl create secret generic efm-db-pass \ --from-literal=password=efm_password \ --namespace cld-streaming kubectl create secret generic efm-encryption \ --from-literal=encryption.password=efm_encryption_key \ --namespace cld-streaming source ~/.env kubectl create secret docker-registry cloudera-registry \ --docker-server=container.repo.cloudera.com \ --docker-username=$CLOUDERA_USER \ --docker-password=$CLOUDERA_PASS \ --namespace=cld-streaming Warning! already exists errors from prior sessions are fine — skip those. 4. Pull the EFM Image into MinikubePermalink eval $(minikube docker-env) docker login container.repo.cloudera.com docker pull container.repo.cloudera.com/cloudera/efm:2.3.1.0-2 Match the tag to your CSO / CEM entitlement. 5. The ConfigMapPermalink Save the following as efm-configMap.yaml. This is the full efm.properties file, and the important part is the efm.db.* block — that is what points EFM at Postgres instead of its default embedded H2 database. apiVersion: v1 kind: ConfigMap metadata: name: efm-config namespace: cld-streaming data: efm.properties: | # Web Server Properties efm.server.address=0.0.0.0 efm.server.port=10090 efm.server.servlet.contextPath=/efm # Cluster Properties efm.cluster.enabled=false # Web Server TLS Properties efm.server.ssl.enabled=false efm.server.ssl.keyStore=./conf/keystore.jks efm.server.ssl.keyStoreType=jks efm.server.ssl.keyStorePassword= efm.server.ssl.keyPassword= efm.server.ssl.trustStore=./conf/truststore.jks efm.server.ssl.trustStoreType=jks efm.server.ssl.trustStorePassword= efm.server.ssl.clientAuth=WANT # User Authentication Properties efm.security.user.auth.enabled=false efm.security.user.auth.adminIdentities=admin efm.security.user.auth.autoRegisterNewUsers=true efm.security.user.auth.authTokenExpiration=12h efm.security.user.auth.groups.manager=INTERNAL efm.security.user.auth.groups.adminIdentities= efm.security.user.auth.groups.filter=.* efm.security.user.certificate.enabled=false efm.security.user.oidc.enabled=false efm.security.user.saml.enabled=false efm.security.user.knox.enabled=false efm.security.user.proxy.enabled=false # Database Properties (PostgreSQL Persistence) efm.db.url=jdbc:postgresql://ssb-postgresql.cld-streaming.svc:5432/efm efm.db.driverClass=org.postgresql.Driver efm.db.username=efm efm.db.password=efm_password efm.db.maxConnections=50 efm.db.sqlDebug=false efm.db.l2CacheEnabled=false # Heartbeat Properties efm.heartbeat.maxAgeToKeep=0 efm.heartbeat.persistContent=false efm.heartbeat.kafka.publishEnabled=false # Edge Event Retention Properties efm.event.cleanupInterval=30s efm.event.maxAgeToKeep.debug=0m efm.event.maxAgeToKeep.info=1h efm.event.maxAgeToKeep.warn=1d efm.event.maxAgeToKeep.error=7d # Agent Class Flow Monitor Properties efm.agentClassMonitor.interval=15s # Agent Monitoring Properties efm.monitor.maxHeartbeatInterval=5m efm.monitor.agentCertExpiryWarningInterval=30d # Operation Properties efm.operation.monitoring.enabled=true efm.operation.monitoring.inQueuedStateTimeoutHeartbeatRate=1.0 efm.operation.monitoring.inDeployedStateTimeout=5m efm.operation.monitoring.inDeployedStateCheckFrequency=1m efm.operation.monitoring.rollingBatchOperationsFrequency=10s efm.operation.monitoring.rollingBatchOperationsSize=100 efm.operation.monitoring.rollingOperationsSize.update.asset=10 efm.operation.monitoring.rollingOperationsSize.update.configuration=100 efm.operation.monitoring.rollingOperationsSize.update.properties=100 efm.operation.monitoring.rollingOperationsSize.sync.resource=10 # Bulletin Registry Properties efm.bulletinregistry.agentBulletinMaxAgeToKeep=5m efm.bulletinregistry.agentClassBulletinMinAgeToKeep=10s efm.bulletinregistry.agentClassBulletinMaxAgeToKeep=5m # Metrics Properties management.metrics.efm.enabled=true management.simple.metrics.export.enabled=false management.prometheus.metrics.export.enabled=true management.prometheus.metrics.export.descriptions=true management.metrics.enable.efm.heartbeat=true management.metrics.enable.efm.repo=true management.metrics.efm.enableTag.host=true management.metrics.efm.enableTag.protocol=false management.metrics.efm.enableTag.agentClass=true management.metrics.efm.enableTag.agentManifestId=true management.metrics.efm.enableTag.agentId=true management.metrics.efm.maxTags.agentClass=20 management.metrics.efm.maxTags.agentManifestId=10 management.metrics.efm.maxTags.agentId=100 management.metrics.tags.application=efm management.metrics.distribution.percentiles.all=.75,.95,.99 # Health and Info Properties efm.actuator.clusterHealthUpdateFrequency=10s efm.actuator.clusterInfoUpdateFrequency=1m management.endpoint.health.showDetails=never management.endpoint.health.showComponents=always management.health.refresh.enabled=false management.health.livenessstate.enabled=false management.health.readinessstate.enabled=false spring.cloud.discovery.client.compositeIndicator.enabled=false # EL Specification Properties efm.el.specifications.dir=./specs # Logging Properties logging.pattern.level=%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}] logging.level.com.cloudera.cem.efm=INFO logging.level.com.hazelcast=WARN logging.level.com.hazelcast.internal.cluster.ClusterService=INFO logging.level.com.hazelcast.internal.nio.tcp.TcpIpConnection=ERROR logging.level.com.hazelcast.internal.nio.tcp.TcpIpConnector=ERROR # General System Settings efm.data.transfer.maxFileSize=16MB efm.data.transfer.cleanupInterval=1h efm.data.transfer.maxAgeToKeep=1d efm.data.transfer.maxEntriesToKeep=100 efm.agentManager.commands.displayLimit=20 spring.main.banner-mode=log efm.asset.s3.downloadRootPath=/tmp/efm-asset-download efm.diagnosticBundle.enabled=false efm.agent-deployer.security.autoConfiguration=false efm.agent-deployer.security.ca.privateKeyPassword= spring.servlet.multipart.max-file-size=100MB spring.servlet.multipart.max-request-size=100MB 6. The Persistent Volume ClaimsPermalink Save the following as efm-pvc.yaml. Two PVCs: one for agent installer binaries, one for uploaded resources like Python scripts and JARs. apiVersion: v1 kind: PersistentVolumeClaim metadata: name: efm-agent-binaries namespace: cld-streaming spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: standard --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: efm-resources namespace: cld-streaming spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: standard Pro Tip! The efm-resources PVC is the one everyone forgets. Without it, uploaded scripts get tracked in the DB but the actual bytes vanish on restart — every flow that references an uploaded resource breaks. 7. The Deployment and ServicePermalink Save the following as efm-deployment-persisted.yaml. This mounts both PVCs, mounts the ConfigMap on top of efm.properties, wires in the secrets as environment variables, and exposes EFM through a LoadBalancer service. apiVersion: apps/v1 kind: Deployment metadata: name: efm namespace: cld-streaming labels: app: efm spec: replicas: 1 selector: matchLabels: app: efm template: metadata: labels: app: efm spec: imagePullSecrets: - name: cloudera-registry containers: - name: efm image: container.repo.cloudera.com/cloudera/efm:2.3.1.0-2 ports: - containerPort: 10090 - containerPort: 9092 env: - name: EF_DB_URL value: "jdbc:postgresql://ssb-postgresql.cld-streaming.svc:5432/efm" - name: EF_REGISTRY_URL value: "http://host.minikube.internal:18080" - name: EF_REGISTRY_ENABLED value: "true" - name: JAVA_OPTS value: "-Dspring.datasource.driver-class-name=org.postgresql.Driver -Def.db.driver.class.name=org.postgresql.Driver" - name: EF_JAVA_OPTS value: "-Dspring.datasource.driver-class-name=org.postgresql.Driver -Def.db.driver.class.name=org.postgresql.Driver" - name: EFM_DB_USER value: efm - name: EFM_DB_PASSWORD valueFrom: secretKeyRef: name: efm-db-pass key: password - name: EFM_ENCRYPTION_PASSWORD valueFrom: secretKeyRef: name: efm-encryption key: encryption.password resources: requests: cpu: "250m" memory: "4Gi" limits: cpu: "250m" memory: "4Gi" volumeMounts: - name: agent-binaries mountPath: /opt/efm/efm-2.3.1.0-2/agent-deployer/binaries - name: efm-resources mountPath: /opt/efm/efm-2.3.1.0-2/resources - name: efm-config mountPath: /opt/efm/efm-2.3.1.0-2/conf/efm.properties subPath: efm.properties readOnly: true volumes: - name: agent-binaries persistentVolumeClaim: claimName: efm-agent-binaries - name: efm-resources persistentVolumeClaim: claimName: efm-resources - name: efm-config configMap: name: efm-config --- apiVersion: v1 kind: Service metadata: name: efm namespace: cld-streaming labels: app: efm spec: type: LoadBalancer ports: - port: 10090 targetPort: 10090 protocol: TCP name: efm-ui - port: 9092 targetPort: 9092 protocol: TCP name: metrics selector: app: efm 8. Apply EverythingPermalink Order matters — ConfigMap and PVCs first, deployment last: kubectl apply -f efm-configMap.yaml -n cld-streaming kubectl apply -f efm-pvc.yaml -n cld-streaming kubectl apply -f efm-deployment-persisted.yaml -n cld-streaming kubectl rollout status deployment/efm -n cld-streaming --timeout=180s Quick sanity check that the ConfigMap actually mounted and that Postgres is in play (not H2): EFM_POD=$(kubectl get pod -n cld-streaming -l app=efm -o jsonpath='{.items[0].metadata.name}') kubectl exec $EFM_POD -n cld-streaming -- sh -c \ 'grep -E "db\.url|db\.driverClass" /opt/efm/efm-2.3.1.0-2/conf/efm.properties' You should see jdbc:postgresql://.... If you see h2, the ConfigMap did not mount — re-apply and restart the deployment. 9. Route the LoadBalancer (Minikube Only)Permalink Minikube needs a little help to assign an external IP. In a separate terminal, run: minikube tunnel Leave it running. If you want the port-forwards and tunnels managed nicely across a workspace, I covered that in my Using Kftray and Zellij post. 10. Access the UIPermalink Open your browser and go to: http://127.0.0.1:10090/efm/ui/ You are in. Create your first agent class, design a flow, publish it, and upload a resource — everything is now backed by Postgres and the two PVCs. 11. Prove the PersistencePermalink The whole point of this exercise. Bounce EFM and confirm nothing disappears: kubectl rollout restart deployment/efm -n cld-streaming kubectl rollout status deployment/efm -n cld-streaming --timeout=180s Refresh the UI. Your agent classes, your flows, and your uploaded resources should all still be there. Agents re-download their assets from the PVC-backed file on the next heartbeat. For an even stronger test, run minikube stop and minikube start. Once EFM’s pod comes back Ready, everything reloads from Postgres and the PVCs automatically. Nothing to re-upload. ResourcesPermalink Cloudera Edge Management (CEM) 2.3.1 Docs MiNiFi C++ Documentation Cloudera Streaming Operators GitHub Repo Using Kftray and Zellij Cloudera Edge Flow Manager on KubernetesPermalink If you would like a deeper dive, hands on experience, demos, or are interested in speaking with me further about Cloudera Edge Flow Manager on Kubernetes, please reach out to schedule a discussion. ... View more

If you are running NiFi Kafka or Flink based applications and workloads in kubernetes, you know that visibility is everything. You can build the most complex data pipelines in the world, but without eyes on your throughput, queues, or other streaming metrics, you’re essentially flying blind. Welcome to the ultimate page for Kubernetes native observability. In this series, we walk through the exact steps to wire up the entire Cloudera Streaming Operator architecture—NiFi, Kafka, and Flink—into a unified Prometheus and Grafana stack. By the end of this journey, you won’t just have basic health checks; you will have a single pane of glass correlating NiFi’s data flow metrics perfectly with Kafka’s topic throughput and Flink’s stream processing metrics. PrerequisitesPermalink This lesson assumes you have already: Completed deployment of Cloudera Streaming Operators Have the minikube branch of Streams Processing Hands on Lab setup completed, nifi flow is running, topics txn,tnx2, and txn_fraud exist, Sql Stream Builder Jobs running with operational polling. Cloned the latest Cloudera Streaming Operators GitHub repo in ~/ local path.  Warning! Some of the excercises include new helm install commands. Be prepared to use your helm uninstall commands as needed. Install/Uninstall is good practice to reset your stage. However if you are using AI to execute against this plan, you can helm upgrade or kubectl apply patches to get desired outcome(s). Prometheus InstallPermalink Before diving into the specific operators, you need to install the central monitoring stack. We will be using the community Prometheus Operator. Ensure your Kubernetes environment is ready, and run the following commands to install the Prometheus Operator and Grafana into the cld-streaming namespace. 1. Add the Helm Repo: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 2. Install the Kube-Prometheus-Stack: This specific configuration enables proxy access, sets up the default datasources, and configures the Operator to watch for PodMonitors and ServiceMonitors across all namespaces ({ }). helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace cld-streaming --create-namespace \ --set grafana.sidecar.datasources.defaultDatasourceEnabled=false \ --set 'grafana.additionalDataSources[0].name=Prometheus' \ --set 'grafana.additionalDataSources[0].type=prometheus' \ --set 'grafana.additionalDataSources[0].url=http://prometheus-kube-prometheus-prometheus.cld-streaming.svc.cluster.local:9090' \ --set 'grafana.additionalDataSources[0].access=proxy' \ --set 'grafana.additionalDataSources[0].isDefault=true' \ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ --set-json 'prometheus.prometheusSpec.serviceMonitorNamespaceSelector={}' \ --set-json 'prometheus.prometheusSpec.podMonitorNamespaceSelector={}' Exposing the Prometheus and Grafana UIsPermalink Grab the URLs and keep the tunnels alive in separate terminals. Tab 1: Prometheus UI minikube service prometheus-kube-prometheus-prometheus -n cld-streaming --url Tab 2: Grafana UI minikube service prometheus-grafana -n cld-streaming --url You can use this command to get the admin password: kubectl get secret --namespace cld-streaming prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo The Cloudera Streaming Operators Integration Series Monitoring Cloudera Streams Messaging (CSM) with Prometheus Monitoring Cloudera Flow Management (CFM) with Prometheus Monitoring Cloudera Streaming Analytics (CSA) with Prometheus 1. Monitoring Cloudera Streams Messaging (CSM) with Prometheus Apache Kafka is the undeniable backbone of modern real-time data, but monitoring its internal health on Kubernetes can often feel like trying to pick a lock. While the Strimzi-powered Cloudera Streams Messaging (CSM) Operator effortlessly spins up your brokers, the critical metrics you need to keep things running smoothly—like byte throughput and under-replicated partitions—are trapped deep inside the JVM. Because Prometheus doesn’t natively speak JMX, we can’t just open a port and call it a day. In Part 1 of this series, we are going to crack open that black box around Kafka. We will walk step-by-step through injecting a custom JMX Prometheus Exporter into your CSM cluster and deploying a specialized PodMonitor to translate those buried JVM metrics into crystal-clear results in Prometheus and Grafana. The Metrics ConfigMapPermalink First, we need to define how Kafka’s JMX metrics are converted into Prometheus format. Create kafka-metrics-config.yaml: kind: ConfigMap apiVersion: v1 metadata: name: kafka-metrics labels: app: strimzi data: kafka-metrics-config.yaml: | # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics lowercaseOutputName: true rules: # Special cases and very specific rules - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" topic: "$4" partition: "$5" - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" broker: "$4:$5" - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections name: kafka_server_$1_connections_tls_info type: GAUGE labels: cipher: "$2" protocol: "$3" listener: "$4" networkProcessor: "$5" - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections name: kafka_server_$1_connections_software type: GAUGE labels: clientSoftwareName: "$2" clientSoftwareVersion: "$3" listener: "$4" networkProcessor: "$5" - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+-total):" name: kafka_server_$1_$4 type: COUNTER labels: listener: "$2" networkProcessor: "$3" - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):" name: kafka_server_$1_$4 type: GAUGE labels: listener: "$2" networkProcessor: "$3" - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+-total) name: kafka_server_$1_$4 type: COUNTER labels: listener: "$2" networkProcessor: "$3" - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+) name: kafka_server_$1_$4 type: GAUGE labels: listener: "$2" networkProcessor: "$3" # Some percent metrics use MeanRate attribute # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate name: kafka_$1_$2_$3_percent type: GAUGE # Generic gauges for percents - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value name: kafka_$1_$2_$3_percent type: GAUGE - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value name: kafka_$1_$2_$3_percent type: GAUGE labels: "$4": "$5" # Generic per-second counters with 0-2 key/value pairs - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count name: kafka_$1_$2_$3_total type: COUNTER labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count name: kafka_$1_$2_$3_total type: COUNTER labels: "$4": "$5" - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count name: kafka_$1_$2_$3_total type: COUNTER # Generic gauges with 0-2 key/value pairs - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value name: kafka_$1_$2_$3 type: GAUGE # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's. # Note that these are missing the '_sum' metric! - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count name: kafka_$1_$2_$3_count type: COUNTER labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" "$6": "$7" quantile: "0.$8" - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count name: kafka_$1_$2_$3_count type: COUNTER labels: "$4": "$5" - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" quantile: "0.$6" - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count name: kafka_$1_$2_$3_count type: COUNTER - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: quantile: "0.$4" # KRaft overall related metrics # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics - pattern: "kafka.server<type=raft-metrics><>(.+-total|.+-max):" name: kafka_server_raftmetrics_$1 type: COUNTER - pattern: "kafka.server<type=raft-metrics><>(current-state): (.+)" name: kafka_server_raftmetrics_$1 value: 1 type: UNTYPED labels: $1: "$2" - pattern: "kafka.server<type=raft-metrics><>(.+):" name: kafka_server_raftmetrics_$1 type: GAUGE # KRaft "low level" channels related metrics # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics - pattern: "kafka.server<type=raft-channel-metrics><>(.+-total|.+-max):" name: kafka_server_raftchannelmetrics_$1 type: COUNTER - pattern: "kafka.server<type=raft-channel-metrics><>(.+):" name: kafka_server_raftchannelmetrics_$1 type: GAUGE # Broker metrics related to fetching metadata topic records in KRaft mode - pattern: "kafka.server<type=broker-metadata-metrics><>(.+):" name: kafka_server_brokermetadatametrics_$1 type: GAUGE Apply the yaml: kubectl apply -f kafka-metrics-config.yaml -n cld-streaming The Kafka Cluster ConfigPermalink Create the kafka-nodepool.yaml: apiVersion: kafka.strimzi.io/v1 kind: KafkaNodePool metadata: name: combined labels: strimzi.io/cluster: my-cluster spec: replicas: 3 roles: - controller - broker storage: type: jbod volumes: - id: 0 type: persistent-claim size: 10Gi kraftMetadata: shared deleteClaim: false Apply the yaml: kubectl apply -f kafka-nodepool.yaml -n cld-streaming` Create the kafka-eval-prometheus.yaml: apiVersion: kafka.strimzi.io/v1 kind: Kafka metadata: name: my-cluster annotations: strimzi.io/node-pools: enabled strimzi.io/kraft: enabled spec: kafka: version: 4.1.1.1.6 metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: kafka-metrics-config.yaml listeners: - name: plain port: 9092 type: internal tls: false - name: tls port: 9093 type: internal tls: true config: offsets.topic.replication.factor: 3 transaction.state.log.replication.factor: 3 transaction.state.log.min.isr: 2 default.replication.factor: 3 min.insync.replicas: 2 entityOperator: topicOperator: {} userOperator: {} Apply the yaml: kubectl apply -f kafka-eval-prometheus.yaml -n cld-streaming Discovery with PodMonitorPermalink Now we tell Prometheus to go find our brokers. Create our PodMoinitor strimzi-pod-monitor.yaml: apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: strimzi-pod-monitor namespace: cld-streaming labels: release: prometheus spec: selector: matchLabels: strimzi.io/cluster: my-cluster strimzi.io/kind: Kafka namespaceSelector: matchNames: - cld-streaming podMetricsEndpoints: - path: /metrics targetPort: 9404 interval: 30s relabelings: # Map Strimzi pod labels (strimzi.io/...) to top-level metric labels the dashboard expects - action: labelmap regex: __meta_kubernetes_pod_label_(strimzi_io_.+) replacement: $1 - action: labelmap regex: __meta_kubernetes_pod_label_(.+) replacement: $1 # Standard K8s labels the dashboard variables use - sourceLabels: [__meta_kubernetes_namespace] targetLabel: namespace - sourceLabels: [__meta_kubernetes_pod_name] targetLabel: kubernetes_pod_name - sourceLabels: [__meta_kubernetes_pod_name] targetLabel: pod_name - sourceLabels: [__meta_kubernetes_pod_node_name] targetLabel: node_name Apply the yaml: kubectl apply -f strimzi-pod-monitor.yaml -n cld-streaming Querying Kafka Metrics in Prometheus UIPermalink You should you have the Prometheus UI exposed via minikube service and your strimzi-pod-monitor shows 3/3 targets UP. Verification: Go to Status -> Targets. Look for strimzi-pod-monitor. It should be UP. Now you can start exploring live metrics from your CSM Operator Kafka cluster in real time. The JMX Prometheus Exporter is successfully scraping your brokers on port 9404. Your Kafka brokers are named my-cluster-combined-* due to the combined KafkaNodePool. In the Prometheus UI switch to the Graph tab, and paste in the queries below. Sample Query 1: Topic Messages In Per Second (Confirmed Throughput) sum(rate(kafka_server_brokertopicmetrics_messagesin_total{topic=~"txn1|txn2|txn_fraud"}[5m])) by (pod, topic) This query aggregates messages ingested per second, grouped by broker pod and topic. Watch it spike when your producers or NiFi flows push data into the txn topics. Excellent for spotting sudden drops or imbalances across brokers. Sample Query 2: Topic Bytes In Per Second (Throughput in Bytes) sum(rate(kafka_server_brokertopicmetrics_bytesin_total[5m])) by (topic) This query shows the incoming byte rate per topic over a 5-minute window. It gives you a clear picture of actual data volume flowing into txn1, txn2, and especially txn_fraud. Because it uses rate(), the graph is much smoother and more useful for monitoring real-world throughput. Quick Tips for This Setup Filter by your actual broker pods when needed: sum(rate(kafka_server_brokertopicmetrics_bytesin_total{namespace="cld-streaming"}[5m])) by (topic, pod) Add namespace filtering for cleaner results: sum(rate(kafka_server_brokertopicmetrics_bytesin_total{namespace="cld-streaming"}[5m])) by (topic) If any query returns no data, make sure you are actively producing messages to the topics. Then restart Prometheus to force a fresh scrape: kubectl rollout restart statefulset prometheus-prometheus-kube-prometheus-prometheus -n cld-streaming Run these sample queries while your NiFi flow is actively sending data to txn1, txn2, and txn_fraud. You should now see clear, live throughput numbers appearing in the Prometheus graphs. This gives you immediate visibility into both message rate and data volume — perfect for evaluating how well your CSM Opeator deployed Kafka cluster is handling the workload. Visualizing CSM Kafka with Grafana DashboardsPermalink With Prometheus feeding live data, Grafana turns those raw metrics into professional dashboards. However, “no data” is a common issue at this stage — usually because Prometheus is not yet scraping the Kafka brokers or the dashboard variables don’t match your labels. Open Grafana (minikube service grafana -n cld-streaming). Login with admin and the password from the secret (see Section 4). Verify the Prometheus Data Source Go to Configuration → Data Sources. The “Prometheus” source should point to something like http://prometheus-operated.monitoring.svc:9090. Click Save & Test. It must say “Data source is working”. (Note: The “Test” button is at the bottom of the datasource edit page.) Import the Cloudera CSM Kafka Dashboard Download the JSON: curl -O https://raw.githubusercontent.com/cldr-steven-matison/ClouderaStreamingOperators/refs/heads/main/csm-kafka-dashboard.json In Grafana → Dashboards → New → Import Click Upload JSON file and select the downloaded file. On the next screen: Datasource → select your Prometheus data source Click Import Boom. You now have the new Cloudera CSM Kafka Dashboard in Grafana: SummaryPermalink With the JMX exporter successfully injected and the PodMonitor active, you have cleared the first major hurdle in building an end-to-end observability pipeline. We didn’t just flip a switch; we architected a robust, Kubernetes-native discovery mechanism that respects the Strimzi-based Operator’s strict validation rules while still providing deep, granular visibility into broker performance. By bridging the gap between Kafka’s internal JMX metrics and Prometheus, you now have the observability needed to monitor everything from message rates to partition health. Whether you are troubleshooting high CPU usage on a specific broker or watching for under-replicated partitions during a scaling event, you now have the raw data required to maintain a healthy cluster. This setup serves as the foundation for the rest of Cloudera Streaming Operator stack. Now that your event backbone (Kafka) is visible, you are ready to plug in your ingestion (NiFi) and processing (Flink) engines to achieve that elusive “single pane of glass” view across the entire data lifecycle in kubernetes. Permalink 2. Monitoring Cloudera Flow Management (CFM) with Prometheus In the previous guide on monitoring Cloudera Streams Messaging (CSM) we added visibility into your Kafka cluster. Data pipelines don’t start at the broker—they often start with NiFi. When running NiFi via the Cloudera Flow Management (CFM) Operator, securing the cluster with Single User Auth puts the APIs into a strict lockdown. This makes scraping native metrics a bit of a kubernetes challenge. In this post, we’re going to wire up a secure CFM NiFi 2.x cluster to the Prometheus + Grafana stack, bypassing web authentication safely using mTLS, and ultimately bridging our cross-namespace metrics into a single pane of glass. The NiFi Cluster Config (The CR)Permalink In NiFi 2.x, Prometheus metrics are built natively into the application; we don’t need an external JMX exporter like Kafka. However, we do need to tell the CFM Operator to disable standard authentication on the metrics endpoint. Update your Nifi Custom Resource (nifi-cluster.yaml) with the configOverride block: apiVersion: [cfm.cloudera.com/v1alpha1](https://cfm.cloudera.com/v1alpha1) kind: Nifi metadata: name: mynifi namespace: cfm-streaming spec: replicas: 1 nifiVersion: "2.6.0" security: initialAdminIdentity: "admin" nodeCertGen: issuerRef: name: cfm-operator-ca-issuer-signed kind: ClusterIssuer singleUserAuth: enabled: true credentialsSecretName: "nifi-admin-creds" configOverride: nifiProperties: upsert: nifi.cluster.leader.election.implementation: "KubernetesLeaderElectionManager" # Disable standard auth for the prometheus endpoint nifi.web.prometheus.metrics.authenticated: "false" Apply this configuration and allow the NiFi pods to perform a rolling restart if necessary. The mTLS VIP Bypass (Finding the Cert)Permalink Because we have singleUserAuth: enabled, NiFi will fiercely defend its endpoints—even with the property override above—throwing 401 Unauthorized errors at Prometheus. NiFi expects a login token. To get around the web login completely, we use Client Certificates (mTLS). The CFM Operator automatically generates a highly privileged cert to talk to NiFi securely. We are going to borrow that cert for Prometheus. Run this command to find the Operator’s user certificate: kubectl get secrets -n cfm-streaming | grep kubernetes.io/tls Look for mynifi-cfm-operator-user-cert. This is our golden ticket which we will take with us below into our NiFi ServiceMonitor. Discovery with ServiceMonitorPermalink Now we tell Prometheus to scrape NiFi, handing it the certificate so it can breeze past the 401 Unauthorized screens. We also use a relabelings block to ensure the Host header perfectly matches what NiFi’s Jetty server expects (preventing a 400 Bad Request error). Save this as nifi-service-monitor.yaml: apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: nifi-service-monitor namespace: cfm-streaming labels: release: prometheus spec: selector: matchLabels: app.kubernetes.io/name: server app.kubernetes.io/instance: mynifi namespaceSelector: matchNames: - cfm-streaming endpoints: - port: https path: /nifi-api/flow/metrics/prometheus interval: 30s scheme: https tlsConfig: insecureSkipVerify: true serverName: mynifi-web.cfm-streaming.svc.cluster.local # Explicit CA mapping fixes the "none configured" error ca: secret: name: mynifi-cfm-operator-user-cert key: ca.crt # The mTLS Bypass Client Certs cert: secret: name: mynifi-cfm-operator-user-cert key: tls.crt keySecret: name: mynifi-cfm-operator-user-cert key: tls.key relabelings: - targetLabel: __address__ replacement: mynifi-web.cfm-streaming.svc.cluster.local:8443 kubectl apply -f nifi-service-monitor.yaml -n cfm-streaming (Wait about 30 seconds. In your Prometheus UI under Status -> Targets, nifi-service-monitor should now show as 1/1 UP). Querying NiFi Metrics in Prometheus UIPermalink Now that Prometheus has a secure, authenticated channel to NiFi, let’s look at the data. Open the Prometheus UI Graph tab and test these queries: Sample Query 1: Total Bytes Queued sum(nifi_amount_bytes_queued{namespace="cfm-streaming"}) Great for setting up alerts if a downstream system (like Kafka) goes offline and backpressure builds up. Sample Query 2: Total Items Queued sum(nifi_amount_items_queued{namespace="cfm-streaming"}) Sample Query 3: Active Threads sum(nifi_active_threads{namespace="cfm-streaming"}) Visualizing CFM NiFi with Grafana DashboardsPermalink With Prometheus pulling the data, let’s load up a beautiful community-built dashboard. Step 1: Import the Dashboard Open Grafana and navigate to Dashboards -> New -> Import. In the “Import via grafana.com” box, type 15822 (or 12375) and click Load. Select your Prometheus data source at the bottom and click Import. Boom. You now have full JVM stats, FlowFile queue tracking, and throughput metrics. SummaryPermalink By leveraging the CFM Operator’s native mynifi-cfm-operator-user-cert, you have successfully engineered an mTLS bridge that bypasses NiFi’s strict Single User Auth lockdown. We didn’t just find a workaround for the “401 Unauthorized” errors; we architected a secure, automated discovery path that allows Prometheus to scrape sensitive metrics without compromising the security of your data orchestration layer. This configuration effectively solves the “networking puzzle” of NiFi 2.x observability. By aligning your ServiceMonitor with NiFi’s strict SNI and Host header requirements, you’ve ensured that your monitoring stack remains as resilient as the pipelines it tracks. You now have the declarative tools to move beyond basic health checks and into deep, cross-namespace correlation. With this piece of the puzzle in place, you can finally realize the “Master Plan”: a single pane of glass where you can watch NiFi’s outbound data rates flow in perfect synchronization with Kafka’s inbound throughput. You no longer have to guess where a bottleneck resides; you have the real-time telemetry required to prove exactly how data is moving through your entire Cloudera Streaming architecture. 3. Monitoring Cloudera Streaming Analytics (CSA) with Prometheus If you followed our previous guides on monitoring Cloudera Streams Messaging (CSM) and Cloudera Flow Management (CFM), you now have visibility into your data ingestion (NiFi) and event streaming (Kafka). But what about monitoring the streams processing jobs (FLINK) in Cloudera Streaming Analytics (CSA)? When running Flink and SQL Stream Builder (SSB) via the CSA Operator, flink jobs spin up dynamically on Kubernetes. Because these dynamically generated TaskManager pods don’t explicitly declare metric ports in their Kubernetes spec, standard Prometheus PodMonitors will silently drop the targets—making job metric discovery a bit of kubernetes spaghetti. In this third and final post of the series, we’re going to wire up our CSA Flink jobs to our existing Prometheus + Grafana stack. By utilizing a Headless Service to bypass strict pod-spec validation natively, we will finally complete plugging our CFM NiFi Operator, CSA Flink Operator, and CSM Kafka Operator into Prometheus and Grafana stack for monitoring. Create the Prometheus Values FilePermalink Create this file in the root of your repo. This forces Flink to open port 9249 for metrics scraping. csa-prometheus-values.yaml # csa-prometheus-values.yaml # Enables native PrometheusReporter for ALL SQL Stream Builder (SSB) jobs ssb: flinkConfiguration: flink-conf.yaml: | metrics.reporters: prom metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory metrics.reporter.prom.port: "9249" taskmanager.network.detailed-metrics: "true" # Optional: cleaner metric labels for Grafana dashboards metrics.scope.jm: "flink.jobmanager.<host>" metrics.scope.tm: "flink.taskmanager.<host>.<tm_id>" metrics.scope.job: "flink.job.<job_id>.<job_name>" Helm Install CommandPermalink Run this exact command: helm install csa-operator \ oci://container.repository.cloudera.com/cloudera-helm/csa-operator/csa-operator \ --namespace cld-streaming \ --create-namespace \ --version 1.5.0-b275 \ --values ./csa-prometheus-values.yaml \ --set 'flink-kubernetes-operator.imagePullSecrets[0].name=cloudera-creds' \ --set 'ssb.sse.image.imagePullSecrets[0].name=cloudera-creds' \ --set 'ssb.sqlRunner.image.imagePullSecrets[0].name=cloudera-creds' \ --set 'ssb.mve.image.imagePullSecrets[0].name=cloudera-creds' \ --set 'ssb.database.imagePullSecrets[0].name=cloudera-creds' \ --set 'ssb.flink.image.imagePullSecrets[0].name=cloudera-creds' \ --set-file flink-kubernetes-operator.clouderaLicense.fileContent=./license.txt Verify the InstallPermalink # 1. Helm release helm list -n cld-streaming # 2. All pods running kubectl get pods -n cld-streaming # 3. Confirm Prometheus config was applied helm get values csa-operator -n cld-streaming | grep -A 20 "flink-conf.yaml" Discovery with Headless Service & ServiceMonitorPermalink Because Flink Native Kubernetes does not explicitly declare port 9249 in its dynamic pod specs, standard PodMonitors will drop the targets. Instead, we bridge the gap using a Headless Service and a ServiceMonitor. A. Create the Headless Service (csa-flink-service.yaml) apiVersion: v1 kind: Service metadata: name: csa-flink-metrics-service namespace: cld-streaming labels: app: csa-flink-metrics spec: clusterIP: None # Makes it a headless service selector: # This automatically captures ALL Flink pods (JobManagers & TaskManagers) type: flink-native-kubernetes ports: - name: prom-metrics port: 9249 targetPort: 9249 B. Create the ServiceMonitor (csa-flink-service-monitor.yaml) apiVersion: [monitoring.coreos.com/v1](https://monitoring.coreos.com/v1) kind: ServiceMonitor metadata: name: csa-flink-metrics-monitor namespace: cld-streaming labels: release: prometheus # Must match your Prometheus Operator release label spec: selector: matchLabels: app: csa-flink-metrics namespaceSelector: matchNames: - cld-streaming endpoints: - port: prom-metrics interval: 15s scrapeTimeout: 10s relabelings: # Extracts labels so Grafana dashboards automatically map deployments - sourceLabels: [__meta_kubernetes_pod_label_app] targetLabel: flink_deployment - sourceLabels: [__meta_kubernetes_pod_label_component] targetLabel: component - sourceLabels: [__meta_kubernetes_pod_name] targetLabel: pod - sourceLabels: [__meta_kubernetes_namespace] targetLabel: namespace C. Apply both files: kubectl apply -f csa-flink-service.yaml -n cld-streaming kubectl apply -f csa-flink-service-monitor.yaml -n cld-streaming Wait ~30 seconds, then check Prometheus UI (Status -> Targets). You should see your JobManagers and TaskManagers listed as UP under serviceMonitor/cld-streaming/csa-flink-metrics-monitor/0. Test Prometheus MetricsPermalink Open SSB UI: minikube service ssb-sse --namespace cld-streaming Run any SQL job in Sql Stream Builder. Verify metrics are exposed directly from a pod: # Replace with your actual taskmanager pod name kubectl exec -it ssb-session-admin-taskmanager-1-3 -n cld-streaming -- \ curl -s http://localhost:9249/metrics | head -20 You should see flink_ metrics. Querying SSB / Flink Metrics in Prometheus UIPermalink Sample Query 1: JVM CPU Load flink_taskmanager_Status_JVM_CPU_Load{namespace="cld-streaming"} Sample Query 2: Job Uptime flink_jobmanager_job_uptime{namespace="cld-streaming"} Sample Query 3: Records In/Out Per Second sum(flink_taskmanager_job_task_operator_numRecordsInPerSecond{namespace="cld-streaming"}) by (job_name) End-to-End Pipeline (NiFi → SSB → Kafka) sum(rate(nifi_bytes_sent{namespace="cfm-streaming"}[5m])) or sum(flink_taskmanager_job_task_operator_numRecordsInPerSecond{namespace="cld-streaming"}) or sum(rate(kafka_server_brokertopicmetrics_bytesin_total{namespace="cld-streaming"}[5m])) Visualizing in GrafanaPermalink Import the Cloudera CSA Flink Dashboard Download the CSA Flink Dashboard JSON: curl -O https://raw.githubusercontent.com/cldr-steven-matison/ClouderaStreamingOperators/refs/heads/main/csa-flink-dashboard.json In Grafana → Dashboards → New → Import Click Upload JSON file and select the downloaded file. On the next screen: Datasource → select your Prometheus data source Click Import Boom. You now have the new Cloudera CSA Flink Dashboard in Grafana: SummaryPermalink With this final piece in place, you have successfully built a complete, end-to-end observability pipeline across your entire Cloudera Streaming Operators architecture. By bridging CFM (NiFi) for ingestion, CSM (Kafka) for event streaming, and CSA (SQL Stream Builder / Flink) for real-time processing, you now have a unified view of your data’s lifecycle within a single Prometheus and Grafana stack. In this specific guide we implemented a Headless Service and a ServiceMonitor to bypass the strict pod-spec limitations of Flink Native Kubernetes. This ensures that every dynamically provisioned JobManager and TaskManager is automatically discovered and scraped by Prometheus, completely eliminating the silent “0 targets” discovery failures during setup. You can now reliably execute complex PromQL queries in Prometheus across namespaces and correlate behavior across entirely different engines. Whether you are tracking backpressure in NiFi, monitoring consumer lag in Kafka, or measuring checkpoint durations and records-per-second in Flink, you finally have the single pane of glass required to confidently debug, tune, scale, and monitor your streaming data pipelines.   End to End CSO Dashboard with GrafanaPermalink Now that we have all of our operator based metrics flowing, all of the operator dashboards setup, and a good understanding of how Prometheus and Grafana queries work. We can easily build a new Fraud Dashboard with Grafana. Download the CSO Fraud Detection Dashboard JSON and import it into Grafana.   Summary: Observability in Kubernetes AchievedPermalink By wiring CFM (NiFi), CSM (Kafka). CSA (Flink/SSB) metrics to Prometheus, you have successfully built the complete, end-to-end observability of the Cloudera Streaming Operators. We didn’t just flip a switch to turn on metrics—we architected a robust, Kubernetes-native solution that respects strict SNI headers, leverages mTLS for secure API scraping, and utilizes headless services to bypass dynamic pod-spec limitations. Best of all, your entire monitoring configuration remains declarative and fully Git-trackable. You can now reliably execute complex PromQL queries across namespaces, correlating behavior across entirely different engines. When you can overlay NiFi’s outbound byte rate directly on top of Kafka’s inbound throughput on the exact same Grafana dashboard, you no longer have to guess where a bottleneck resides. You have the telemetry to prove it. ResourcesPermalink Cloudera Streams Messaging (CSM) 1.6 Docs Cloudera Streaming Analytics (CSA) 1.5 Docs Cloudera Flow Management (CFM) 3.0 Docs Cloudera Streaming Operators GitHub Repo Cloudera Streaming Operators Blog ... View more