Created on - edited on by
K23
Ambari Metric System (AMS) HA
Ambari Metrics System is an Ambari-native pluggable and scalable system for collecting and querying Hadoop Metrics, that includes Grafana, a powerful dashboard builder that is fully open source with a wide community adoption. By default, Metrics Collector is the REST API component that receives metrics payload as JSON over HTTP from the Sinks and Monitors. The metrics are written into the HBase storage layer which is dedicated storage for metric data and managed as a part of AMS, separate from the cluster HBase. The HBase schema is defined using Phoenix and all the read write operations from AMS are Phoenix jdbc API calls. The Sink implementations are native to AMS and are placed in the classpath of the supported Hadoop ecosystem services by Ambari. The Monitors are lightweight python daemons for system counters that use psutil native libraries for data collection. AMS can scale horizontally by adding additional Collector nodes which effectively adds additional HBase Regionserver(s) to handle increased read/write load. Ambari stack advisor is utilized to advise on AMS configurations proportional to the number of Sinks and monitors and thereby the cluster size.
For this article, the CA has provided with a couple of PKCS#12 bundle of certificates called amc01.p12 and amc02.p12. Since is the same CA for both certificates, from one of them, you will get the CA certificates (root + Intermediates).
This configuration assumes the following locations:
- /var/tmp/certificates/AMS. The path where will be copied the PKCS#12 bundles.
- /var/tmp/certificates/AMS/TRUSTSTORE. The path where will be created the truststore for all nodes .
- /var/tmp/certificates/AMS/KEYSTORE/{AMC01,AMC02}. The path to create the keystore for the collectors.
- /usr/jdk64/jdk1.8.0_112. The path for the java version installed.
- c3132-node1, c3132-node2, c3132-node3, c3132-node4. HDP Cluster Nodes.
- c3132-node1. Ambari Server.
- c3132-node2, c3132-node3. Cluster nodes configured as Ambari Metrics Collectors.
- /labs/AMS/truststore.jks. The path for truststore in all nodes.
- /labs/AMS/keystore.jks. The path for keystore in each of Ambari Metrics Collector.
SSL Setup Logical Steps
Basically, for each metrics collector, add the PKCS#12 bundle identified by an alias with the Metrics Collector FQDN with PrivateKeyEntry, and the RootCA and Intermediate certificates in a Truststore identified by an alias with trustedCertEntry.
Every time Ambari starts the service, it will try to export the rootCA and intermediate certificates from the Truststore located in all nodes. First, it will try converting the Truststore from JKS format to PKCS12 format, then exporting all the CA certificates from the Truststore to its configuration directory creating the file called ca.pem. You could see the following messages from Ambari Operations Status Page.
Execute['ambari-sudo.sh /usr/jdk64/jdk1.8.0_112/bin/keytool -importkeystore -srckeystore
/labs/AMS/truststore.jks -destkeystore /tmp/tmp0_1xE1/truststore.p12
-srcalias c3132-node3.user.local -deststoretype PKCS12 -srcstorepass hadoop1234
-deststorepass hadoop1234'] {}
Execute['ambari-sudo.sh /usr/jdk64/jdk1.8.0_112/bin/keytool -importkeystore -srckeystore
/labs/AMS/truststore.jks -destkeystore /tmp/tmp0_1xE1/truststore.p12 -srcalias
c3132-node2.user.local -deststoretype PKCS12 -srcstorepass hadoop1234 -deststorepass hadoop1234'] {}
Execute['ambari-sudo.sh openssl pkcs12 -in /tmp/tmpI3YmtL/truststore.p12 -out
/etc/ambari-metrics-monitor/conf/ca.pem -cacerts -nokeys -passin pass:hadoop1234'] {}
Follow these steps to complete the previous setup. For this procedure, the node c3132-node2.user.local will hold the Active Ambari Metrics Collector.
- Since you received a couple of certificates bundle from the same Certificate Authority, you will Extract CA Certificates from one of the PKCS#12 Bundle
cd /var/tmp/certificates/AMS && ls -l openssl pkcs12 -in c3132-node2.user.local.p12 -out rootca.crt -cacerts -nokeys -passin pass:hadoop1234
- Create the truststore and add the CA Certificate.
/usr/jdk64/jdk1.8.0_112/bin/keytool -keystore TRUSTSTORE/truststore.jks -alias caroot -import -file rootca.crt -storepass hadoop1234 /usr/jdk64/jdk1.8.0_112/bin/keytool -list -keystore TRUSTSTORE/truststore.jks
- Add to the truststore the PrivateCertEntry for all the Ambari Metrics Collectors using the FQDN as an alias
/usr/jdk64/jdk1.8.0_112/bin/keytool -importkeystore -srckeystore c3132-node2.user.local.p12 -alias c3132-node2.user.local -destkeystore TRUSTSTORE/truststore.jks -srcstoretype pkcs12 -deststoretype jks /usr/jdk64/jdk1.8.0_112/bin/keytool -importkeystore -srckeystore c3132-node3.user.local.p12 -alias c3132-node3.user.local -destkeystore TRUSTSTORE/truststore.jks -srcstoretype pkcs12 -deststoretype jks /usr/jdk64/jdk1.8.0_112/bin/keytool -list -keystore TRUSTSTORE/truststore.jks
- Create the keystore for the first Ambari Metrics Collector adding the rootca as a TrustedCertEntry and server as a PrivateKeyEntry
/usr/jdk64/jdk1.8.0_112/bin/keytool -keystore KEYSTORE/AMC01/keystore.jks -alias caroot -import -file rootca.crt -storepass hadoop1234 /usr/jdk64/jdk1.8.0_112/bin/keytool -importkeystore -srckeystore c3132-node2.user.local.p12 -alias c3132-node2.user.local -destkeystore KEYSTORE/AMC01/keystore.jks -srcstoretype pkcs12 -deststoretype jks
- Create the keystore for the second Ambari Metrics Collector adding the rootca as a TrustedCertEntry and server as a PrivateKeyEntry
/usr/jdk64/jdk1.8.0_112/bin/keytool -keystore KEYSTORE/AMC02/keystore.jks -alias caroot -import -file rootca.crt -storepass hadoop1234 /usr/jdk64/jdk1.8.0_112/bin/keytool -importkeystore -srckeystore c3132-node3.user.local.p12 -alias c3132-node3.user.local -destkeystore KEYSTORE/AMC02/keystore.jks -srcstoretype pkcs12 -deststoretype jks
- Copy the truststore to all nodes, including Ambari server and the keystore for each Ambari Metrics Collector
for i in c3132-node1 c3132-node2 c3132-node3 c3132-node4
do
ssh root@${i} "mkdir -p /labs/AMS"
scp /var/tmp/certificates/AMS/TRUSTSTORE/truststore.jks root@${i}:/labs/AMS/
if [[ ${i} == "c3132-node2" ]] ; then
scp /var/tmp/certificates/AMS/KEYSTORE/AMC01/keystore.jks root@${i}:/labs/AMS/
elif [[ ${i} == "c3132-node3" ]] ; then
scp /var/tmp/certificates/AMS/KEYSTORE/AMC02/keystore.jks root@${i}:/labs/AMS/
else
echo
fi
done
- From Ambari, configure the SSL properties (SSL Server/Client) to reference the Keystore and Truststore.
AmbariUI / Services / Ambari Metrics / Configs / ams-site timeline.metrics.service.http.policy=HTTPS_ONLY ams-ssl-server ssl.server.keystore.keypassword=hadoop1234 ssl.server.keystore.location=/labs/AMS/keystore.jks ssl.server.keystore.password=hadoop1234 ssl.server.keystore.type=jks ssl.server.truststore.location=/labs/AMS/truststore.jks ssl.server.truststore.password=hadoop1234 ssl.server.truststore.reload.interval=10000 ssl.server.truststore.type=jks ams-ssl-client ssl.client.truststore.location=/labs/AMS/truststore.jks ssl.client.truststore.password=hadoop1234 ssl.client.truststore.type=jks AmbariUI -> Restart All Required
- Configure Ambari server to use https instead of http in all the requests to AMS Collector
ssh root@c3132-node1 echo "server.timeline.metrics.https.enabled=true" >> /etc/ambari-server/conf/ambari.properties ambari-server setup-security Using python /usr/bin/python Security setup options... =========================================================================== Choose one of the following options: [1] Enable HTTPS for Ambari server. [2] Encrypt passwords stored in ambari.properties file. [3] Setup Ambari kerberos JAAS configuration. [4] Setup truststore. [5] Import certificate to truststore. =========================================================================== Enter choice, (1-5): 4 Do you want to configure a truststore [y/n] (y)? y TrustStore type [jks/jceks/pkcs12] (jks): Path to TrustStore file :/labs/AMS/truststore.jks Password for TrustStore: Re-enter password: Ambari Server 'setup-security' completed successfully. ambari-server restart
- From one of the Ambari Metrics Monitor validate the https comunnication.
ssh root@c3132-node4 tail -f /var/log/ambari-metrics-monitor/ambari-metrics-monitor.log
The following messages reflects HTTPS communication to the active Metrics Collector:
2018-12-12 02:27:11,835 [INFO] emitter.py:210 - Calculated collector shard based on hostname : c3132-node2.user.local 2018-12-12 02:27:11,835 [INFO] security.py:52 - SSL Connect being called.. connecting to https://c3132-node2.user.local:6188/ 2018-12-12 02:27:11,855 [INFO] security.py:43 - SSL connection established.
