Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

BDR Peer failed after SSL lv1 with wildcard certificate

Highlighted

BDR Peer failed after SSL lv1 with wildcard certificate

New Contributor

Hi all, I'm working on 2 cluster CDH 5.13 with one-way replication using BDR which already worked and scheduled. after implementation of SSL lv1 on both CM and Hue, replication schedule failed with

error : Unable to find source service. Verify peer connectivity and service.

 After check the peer connection, it pops :

Unknown exception of type javax.ws.rs.RedirectionException while connecting to dest_cluster_address

I suspect it needs to be changed to https and port 7183, but when i edit the connectivity to secure port and address, the error while saving were : 

javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://destCMhostname:7183/api/v1/users: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

the SSL itself were using wildcard signed certificate which used on both cluster since it shares the same domain so assume the jssecacerts truststore for both CM already has the same PEM certificate imported.

 

Also tried to exchange certificate of both cluster and import to each jssecacerts truststore and restart CM service from OS RHEL 7.2 on both cluster but peer connection still failed to connect. Both cluster has kerberized with its own KDC server that have trust-realm established also.

 

Anyone have workaround or further steps of investigation?

 

Thanks Al

4 REPLIES 4

Re: BDR Peer failed after SSL lv1 with wildcard certificate

Super Collaborator

Hi @alcarin_ducil ,

 

Please take a look at this documentation:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_bdr_replication_source.html

 

Especially below notes:

Cloudera recommends that TLS/SSL be used. A warning is shown if the URL scheme is http instead of https. After configuring both peers to use TLS/SSL, add the remote source Cloudera Manager TLS/SSL certificate to the local Cloudera Manager truststore, and vice versa.

 

Thanks and hope this helps.

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Highlighted

Re: BDR Peer failed after SSL lv1 with wildcard certificate

New Contributor

Hi @lwang 

 

Thanks for the reply..

Yes, i do note the necessity to add source CM TLS/SSL certificate to destination truststore and vice versa. does this certificate means PEM files or JKS files? i used the same wildcard PEM files for configure TLS/SSL for both cluster since both source and destination are on same domain.

When i import PEM from source CM to destination CM truststore (jssecacerts), it stated certificate already exist but i import it anyway under different alias [also done it vice versa on source CM truststore].  After restart cloudera-scm-server  on both cluster, peer connection still failed.

 

Does configuring BDR Peer can be using same wildcard for TLS/SSL connection between 2 cluster?

 

Thanks,

Al

Highlighted

Re: BDR Peer failed after SSL lv1 with wildcard certificate

Super Guru

Hi @alcarin_ducil ,

 

To answer one question:  Cloudera applications that are written in java use a Java Keystore that is in the JKS format; Cloudera Manager is a Java application so it uses a JKS file to determine trust in TLS handshakes. 

Based on the error snippet you supplied, it appears that the operation whereby the destination cluster's Cloudera Manager instance attempted to make a connection to the source cluster's Cloudera Manager.  When doing so, the TLS handshake failed and the following was presented:

 

javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://destCMhostname:7183/api/v1/users: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

 

In this particular case, the BDR destination (target) cluster Cloudera Manager server must trust the signer of the certificate presented by the source Cloudera Manager; as the error indicates, this trust could not be found.

 

One possible cause of this situation is that Cloudera Manager is using a file as a store for certificate signers that does not contain trust for the signer.  Cloudera Manager will use Administration --> Settings --> Security --> Cloudera Manager TLS/SSL Client Trust Store File unless it is not configured.  If it is not configured, it will use the JDK's jssecacert; if that does not exist, then the default cacert file is used.  See the following for more background:
https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustMana...

 

I would recommend checking (using keytool -list -v -keystore /path/to/keystore/file) to make sure you have the signing certificate for the source cluster in the JKS files.

The key is understanding the chain of trust.  Are you using a self-signed certificate (where the signer is the same as the subject (in which case, yes, you should be able to use the same certificates for the server and clients.  If you used another certificate authority to sign your server certificates, however, you will need to add that signer's public certificate to the truststores.

 

Let's start there and see if that helps.  If not, please let us know.

 

Regards,

 

Ben

Highlighted

Re: BDR Peer failed after SSL lv1 with wildcard certificate

New Contributor

Hi @bgooley ! thanks for replying back.

sorry for the delay since this COVID-19 situation i can't get to the cluster for troubleshooting steps until now. To follow up the test using the keystore and truststore of each cluster CM, i found that JKS files in both clusters CM are contains 2 entries : PrivateKeyEntry and trustedCertEntry with Certificate fingerprint (SHA1) completely identical, only differs in alias name

Target BDR JKS content:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

servercert, Dec 20, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE
cmdrc.company.co.id, Dec 20, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE

 

Source BDR JKS content:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

servercert, Jan 16, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE
cmprod.company.co.id, Jan 16, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE

While on truststore on each cluster CM, entries with same Certificate fingerprint (SHA1) were found also :

Target BDR truststore jssecacerts content:

rootca, Jan 16, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE

 

Source BDR truststore jssecacerts content:

rootca2, Dec 20, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE

 

For another background, the process of setting TLS/SSL on both cluster are a bit different from Cloudera docs. Since the cluster share the same domain and security team already had signed certificate from trusted CA, we  were provided with signed wildcard certificate in .PEM and . KEY files. so instead of generate JKS file from Java keytool on Cloudera nodes, we executes this steps :

  1. Create p12 file by combining signed certificate CA from root to local PEM files with corresponding KEY files (openssl pkcs12 -export command)
  2. Convert p12 file to JKS file with name of each hostname (keytool -importkeystore -destkeystore command)
  3. Import the root CA certificate into the JDK truststore jssecacerts (keytool -importcert command)

Also another question, is BDR are sufficient with Level 1: Encryption only TLS/SSL setup on both cluster or is it require more (Level 2 :server-side certificate validation or even Level 3: dual certificate validation)?

 

Many thanks,

Al

Don't have an account?
Coming from Hortonworks? Activate your account here