With the need of keeping metrics centralized into a single spot, people have sought a way to configure grafana for observability by attempting to collect the wide range of metrics available in a Datahub / Datalake in order to keep their dashboards arranged in a centralized hub. This tutorial will walk you through step-by-step in how to configure Grafana to query the metrics available in the Cloudera Manager of a datahub cluster.
First, we need to ensure that the machine where Grafana is running has direct connection with the CM Server. In other words, it must be able to resolve the CM FQDN and establish a straight up connection with the service, bypassing knox completely.
Unlike the Grafana utilized by other experiences, such as DWX, CDW and so forth of which use prometheus for integration, this grafana will have to integrate with CM Server for authentication and during the creation of a datasource when querying the metrics therein. This integration is available through a plugin that must be installed after the grafana deployment, which are by no means maintained or developed by Cloudera.
Installing Grafana:
* yum -y install grafana
* systemctl start grafana-server
Make sure the service has proper access to the folder "/var/lib/grafana/""
* grafana-cli plugins install foursquare-clouderamanager-datasource
Grafana server must be restarted prior to utilizing the plugins.
* systemctl restart grafana-server
The first step is to locate the machine whose CM Server is running and fetch the appropriate client certificate in use by the service, and this step can be accomplished from any server with proper access to the Cloudera Manager.
Ex.
openssl s_client -showcerts -connect <datahub-name>.dperez-a.a465-9q4k.cloudera.site:7183
the port 7183 is the secured HTTPS endpoint used by CM, therefore we can use openssl in order to extract the client certificate actively used by the service, widely adopted by many sysadmins when having to deal with TLS/SSL circumstances.
As a next step, we can toggle on the option "With CA Cert" and paste the certificate acquired as part of the previous step in addition with the workload user and password under "Basic Auth Details"
The format of the URL should follow the pattern down below, where the CM_SERVER_FQDN must be replaced with the appropriate CM Server in use by your Datahub. If there are any errors along the way, you should see a pop up message displaying the exact error message. Furthermore, you can always inspect the grafana logs for further clarifications if the error isn't intuitive at first sight.
https://CM_SERVER_FQDN:7183
After the procedure is completed, you should be able to start setting up the charts by providing a valid tsquery. The tsquery language is used to specify statements for retrieving time-series data from the Cloudera Manager time-series datastore. You can check the tsquery of those many charts available in your CM UI and use it as a reference when building your own set of charts.
Ref: https://grafana.com/grafana/plugins/foursquare-clouderamanager-datasource/?tab=installation
Ref: https://github.com/foursquare/datasource-plugin-clouderamanager
Ref: https://docs.cloudera.com/cloudera-manager/7.4.2/metrics/topics/cm-metrics-reference.html
