Support Video: How to Configure Grafana to use CA Signed Certificates?

[#1]

This articles describe the steps required to complete the setup for accessing Grafana using HTTPS with CA Signed Certificates.

Open YouTube video here

Ambari Metrics System includes Grafana, which is a daemon that runs on a specific host in the cluster and serves pre-built dashboards for visualising metrics collected in the Metrics Collector. 

For this article the following servers is used:

• 172.25.33.152 c3132-node1.user.local (Ambari Server)
• 172.25.36.9   c3132-node2.user.local (Ambari Metrics Collector + Grafana)
• 172.25.40.27  c3132-node3.user.local (Ambari Metrics Collector)
• 172.25.33.163 c3132-node4.user.local

By default, Grafana listen on  port TCP/3000:

# for i in $(netstat -utnlp | awk '/grafana/ {print substr($7, 1, length($7)-13)}' | 
sort -u) ; do echo ; ps -eo pid,user,command --cols 128 | grep $i | grep -v grep ; 
netstat -utnlp | grep $i ; echo ; done

270925 ams      /usr/lib/ambari-metrics-grafana/bin/grafana-server 
--pidfile=/var/run/ambari-metrics-grafana/grafana-server.pid
tcp6       0      0 :::3000                 :::*                   
 LISTEN      270925/grafana-serv

Here, the process running is grafana-server, the owner is ams, and is listening on port TCP/3000. All the configurations for Grafana are handled by Ambari, and are reflected in the ams-grafana.ini file located at /etc/ambari-metrics-grafana/conf/ directory. Grafana needs to be restarted for any configuration changes to take effect.

In enterprises where security is required, limit the Grafana access to only HTTPS connections. To enable https for Grafana, update the following properties:

AmbariUI / Services / Ambari Metrics / Configs -> Advanced ams-grafana-ini

• protocol: By default, http. For this video we need to change this to https.
• ca_cert: The path to CA root certificate or bundle to be used to validate the Grafana certificate against. Since we are using a PKCS#12 bundle certificate, we need to extract the CA certificate chain from it. 
• cert_file: The path to the certificate. This certificate nees to be in PEM format. 
• cert_key: The path for the private key that match with the public key of the certificate. This private key needs to be unencrypted RSA private key.

For this article, the CA will provide us with a certificate bundle located at:

/var/tmp/certificates/GRAFANA

Since the certificate information provided by the CA is a PKCS#12 certificate bundle, complete the following steps:

1. Extract the root and intermediate certificates, using the following command:

   openssl pkcs12 -in c3132-node2.user.local.p12 -out ams-ca.crt -cacerts -nokeys 
   -passin pass:hadoop1234

2. Extract the server certificate:

   openssl pkcs12 -in c3132-node2.user.local.p12 -out ams-grafana.crt -clcerts 
   -nokeys -passin pass:hadoop1234

3. Extract the private key:

   openssl pkcs12 -in c3132-node2.user.local.p12 -nocerts -nodes -out ams-grafana.key 
   -passin pass:hadoop1234

4. Copy the certificates to a folder with ams user permissions. For this article, the default path and the default names the following:

   cp ams-*.* /etc/ambari-metrics-grafana/conf/
   chown ams:hadoop /etc/ambari-metrics-grafana/conf/ams-*.*

5. Update the Grafana configuration from Ambari: 

   AmbariUI / Services / Ambari Metrics / Configs -> Advanced ams-grafana-ini
   
   protocol = https
   ca_cert = /etc/ambari-metrics-grafana/conf/ams-ca.crt
   cert_file = /etc/ambari-metrics-grafana/conf/ams-grafana.crt
   cert_key = /etc/ambari-metrics-grafana/conf/ams-grafana.key

6. Save the changes, and restart all affected services.
7. Double-check the Grafana log file:

   tail -f /var/log/ambari-metrics-grafana/grafana.log

8. The following will be the listening on port 3000 over HTTPS: 

   2018/12/12 03:42:41 [I] Listen: https://0.0.0.0:3000

9. Double-check the certificate in place using:

   openssl s_client -connect c3132-node2.user.local:3000  </dev/null

10. Open Grafana from Ambari to validate if its working as expected:

    AmbariUI / Services / Ambari Metrics / Summary -> Quick Link Grafana

    Note: Ignore the warning and proceed.

With all these steps,  Grafana is configured to be used with  CA Signed certificates and the communication will be is over HTTPS.