BDR Peer failed after SSL lv1 with wildcard certificate

[#1]

Hi all, I'm working on 2 cluster CDH 5.13 with one-way replication using BDR which already worked and scheduled. after implementation of SSL lv1 on both CM and Hue, replication schedule failed with

error : Unable to find source service. Verify peer connectivity and service.

 After check the peer connection, it pops :

Unknown exception of type javax.ws.rs.RedirectionException while connecting to dest_cluster_address

I suspect it needs to be changed to https and port 7183, but when i edit the connectivity to secure port and address, the error while saving were : 

javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://destCMhostname:7183/api/v1/users: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

the SSL itself were using wildcard signed certificate which used on both cluster since it shares the same domain so assume the jssecacerts truststore for both CM already has the same PEM certificate imported.

Also tried to exchange certificate of both cluster and import to each jssecacerts truststore and restart CM service from OS RHEL 7.2 on both cluster but peer connection still failed to connect. Both cluster has kerberized with its own KDC server that have trust-realm established also.

Anyone have workaround or further steps of investigation?

Thanks Al

[#2]

Hi @alcarin_ducil ,

Please take a look at this documentation:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_bdr_replication_source.html

Especially below notes:

Cloudera recommends that TLS/SSL be used. A warning is shown if the URL scheme is http instead of https. After configuring both peers to use TLS/SSL, add the remote source Cloudera Manager TLS/SSL certificate to the local Cloudera Manager truststore, and vice versa.

Thanks and hope this helps.

[#3]

Hi @lwang 

Thanks for the reply..

Yes, i do note the necessity to add source CM TLS/SSL certificate to destination truststore and vice versa. does this certificate means PEM files or JKS files? i used the same wildcard PEM files for configure TLS/SSL for both cluster since both source and destination are on same domain.

When i import PEM from source CM to destination CM truststore (jssecacerts), it stated certificate already exist but i import it anyway under different alias [also done it vice versa on source CM truststore].  After restart cloudera-scm-server  on both cluster, peer connection still failed.

Does configuring BDR Peer can be using same wildcard for TLS/SSL connection between 2 cluster?

Thanks,

Al

[#4]

Hi @alcarin_ducil ,

To answer one question:  Cloudera applications that are written in java use a Java Keystore that is in the JKS format; Cloudera Manager is a Java application so it uses a JKS file to determine trust in TLS handshakes. 

Based on the error snippet you supplied, it appears that the operation whereby the destination cluster's Cloudera Manager instance attempted to make a connection to the source cluster's Cloudera Manager.  When doing so, the TLS handshake failed and the following was presented:

javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://destCMhostname:7183/api/v1/users: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

In this particular case, the BDR destination (target) cluster Cloudera Manager server must trust the signer of the certificate presented by the source Cloudera Manager; as the error indicates, this trust could not be found.

One possible cause of this situation is that Cloudera Manager is using a file as a store for certificate signers that does not contain trust for the signer.  Cloudera Manager will use Administration --> Settings --> Security --> Cloudera Manager TLS/SSL Client Trust Store File unless it is not configured.  If it is not configured, it will use the JDK's jssecacert; if that does not exist, then the default cacert file is used.  See the following for more background:
https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustMana...

I would recommend checking (using keytool -list -v -keystore /path/to/keystore/file) to make sure you have the signing certificate for the source cluster in the JKS files.

The key is understanding the chain of trust.  Are you using a self-signed certificate (where the signer is the same as the subject (in which case, yes, you should be able to use the same certificates for the server and clients.  If you used another certificate authority to sign your server certificates, however, you will need to add that signer's public certificate to the truststores.

Let's start there and see if that helps.  If not, please let us know.

Regards,

Ben

[#5]

Hi @bgooley ! thanks for replying back.

sorry for the delay since this COVID-19 situation i can't get to the cluster for troubleshooting steps until now. To follow up the test using the keystore and truststore of each cluster CM, i found that JKS files in both clusters CM are contains 2 entries : PrivateKeyEntry and trustedCertEntry with Certificate fingerprint (SHA1) completely identical, only differs in alias name

Target BDR JKS content:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

servercert, Dec 20, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE
cmdrc.company.co.id, Dec 20, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE

Source BDR JKS content:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

servercert, Jan 16, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE
cmprod.company.co.id, Jan 16, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE

While on truststore on each cluster CM, entries with same Certificate fingerprint (SHA1) were found also :

Target BDR truststore jssecacerts content:

rootca, Jan 16, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE

Source BDR truststore jssecacerts content:

rootca2, Dec 20, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE

For another background, the process of setting TLS/SSL on both cluster are a bit different from Cloudera docs. Since the cluster share the same domain and security team already had signed certificate from trusted CA, we  were provided with signed wildcard certificate in .PEM and . KEY files. so instead of generate JKS file from Java keytool on Cloudera nodes, we executes this steps :

1. Create p12 file by combining signed certificate CA from root to local PEM files with corresponding KEY files (openssl pkcs12 -export command)
2. Convert p12 file to JKS file with name of each hostname (keytool -importkeystore -destkeystore command)
3. Import the root CA certificate into the JDK truststore jssecacerts (keytool -importcert command)

Also another question, is BDR are sufficient with Level 1: Encryption only TLS/SSL setup on both cluster or is it require more (Level 2 :server-side certificate validation or even Level 3: dual certificate validation)?

Many thanks,

Al