Hive view crashing on ambari after setting up SSL (no KERBEROS)

[#1]

Hi folks, Recently I used the following script to setup SSL on a 3 node Ambari cluster : https://community.hortonworks.com/articles/22756/quickly-enable-ssl-encryption-for-hadoop-component....

That was really useful, but now ambari view crashes with error "RA040 I/O error while requesting Ambari "

I think that somehow, the script didn't setup properly the hive service, and something with the certificates went wrong.., its like if hive doesnt like the generated certificates, (which actually I recreated following other forum ideas) Perhaps I have to do it manualley, but honestly, I dont know how to do that.. and I am getting quite crazy..

Could you please help me with that??

Here is the trace of the error:

Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching master1.pf0g2dnjye1ujcvq5102dppltf.ax.internal.cloudapp.net found
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
	at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:209)
	at org.apache.ambari.server.view.ViewAmbariStreamProvider.getInputStream(ViewAmbariStreamProvider.java:118)
	at org.apache.ambari.server.view.ViewAmbariStreamProvider.readFrom(ViewAmbariStreamProvider.java:78)
	at org.apache.ambari.view.utils.ambari.URLStreamProviderBasicAuth.readFrom(URLStreamProviderBasicAuth.java:65)
	at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:173)
	... 97 more
Caused by: java.security.cert.CertificateException: No name matching master1.pf0g2dnjye1ujcvq5102dppltf.ax.internal.cloudapp.net found
	at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:221)
	at sun.security.util.HostnameChecker.match(HostnameChecker.java:95)
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:200)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)

[#2]

@david garcia

You are getting the error:

Caused by: java.security.cert.CertificateException: No name matching master1.xxx.yyyy.net found 

Please check the Certificate CN (Common Name) that you used to create the certificate. It should be the same as hostname teing used in the URL while accessing the service component. If the CN in the certificate is not the same as the hostname of the component, you will get the error "java.security.cert.CertificateException: No name matching <host_name> found", where , "host_name" is the host name you used as part of the URL in your client application. (Here client is hive view)

.

Please not that, you cannot change the CN in an already created certificate. So you will need to create a new certificate with the correct hostname of the service.

[#3]

Hi @Jay SenSharma, First of all, thanks for your attention, I recreated the certificates (actually I changed the script I used to create them) and now

CN=master1.pf0g2dnjye1ujcvq5102dppltf.ax.internal.cloudapp.net, as you can see in the following code

keytool -list -storepass changeit -keystore /etc/pki/java/cacerts

Keystore type: JKS
Keystore provider: SUN
Your keystore contains 2 entries

myownca, Jul 19, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): 15:B5:29:D2:B7:CE:53:E4:88:FD:2F:11:FF:2A:E9:D4:D9:C3:97:6E
master1.pf0g2dnjye1ujcvq5102dppltf.ax.internal.cloudapp.net, Jul 19, 2017, trustedCertEntry,
Certificate fingerprint (SHA1): 08:11:7D:AB:D9:C1:16:02:AF:FB:E2:4A:32:1F:E9:43:C9:43:78:B1

These certificates are stored in /etc/pki/java/cacerts in my main master node (master1) , and the hiveserver2 im connecting to, is in the master2 node, should I copy the certificates or change something aditionally there to allow the connection? I think Ambari does atomatically this.. doesnt it? I additionally saw something about changing some custom hive-site conf values , and I added variables with these values:

hive.server2.keystore.path =   /etc/pki/java/cacerts    
hive.server2.trustore.path  =   /etc/pki/java/cacerts
hive.server2.use.SSLhive.server2.use.SSL = true
ssl.trustStore.type     = jks

Cant the truststore and keystore be the same?

but I am still getting the same handshake error..

Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching master1.pf0g2dnjye1ujcvq5102dppltf.ax.internal.cloudapp.net found

Thank you very much for your help, I am quite frustrated at this point..

[#4]

Hi @jay SenSharma

Just found that ssh connection is not possible on console between master1 and master2, and maybe its related: Should I add explicitly any key to master2 ? I have tried to read documentation but the link which appears on the offcial documentation to setup ssh seems not work:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Installing_HDP_AMB/content/_set_up_passw...

 ssh -v 172.31.0.5
Cconnectng to 172.31.0.5 [172.31.0.5] port 22.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 33:41:90:c6:84:47:de:6a:95:b4:1a:2f:1d:f5:88:3d
debug1: Host '172.31.0.5' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available


debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available


debug1: Unspecified GSS failure.  Minor code may provide more information




debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available


debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)

may

[#5 · Accepted solution]

To resolve the issue, import the Ambari certificates to the Ambari truststore. To import the Ambari certificates, do the following: 

STEP 1: 
Get certificate from ambari-server 
echo | openssl s_client -showcerts -connect <AMBARI_HOst>:<AMBARI_HTTPs_PORT> 2>&1 | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/ambari_certificate.cr 

STEP 2: 
Get path of ambari trustore and truststore password from Ambari properties 
cat /etc/ambari-server/conf/ambari.properties |grep truststore 

As per your ambari.properties below is the path and password :-
ssl.trustStore.password=refer from ambari.property file
ssl.trustStore.path=/etc/ambari-server/conf/ambari-server-truststore

STEP 3: 
keytool -importcert -file /tmp/ambari_certificate.crt -keystore <keystore-path> 

STEP 4: 
ambari-server restart