SSL handshake failure for AWS-hosted parcel repositories

[#1]

We have a parcel repository hosted in Amazon S3, and also have it configured to be accessible via https using AWS Certificate Manager.  The certificate is valid, and other tools (curl, Chrome) have no issues accessing the repository via SSL.

When adding the repository to Cloudera Manager with an https:// prefix, it fails with an SSL handshake failure (below).  Are SSL-enabled custom parcel repositories supported?  Observed on Cloudera Manager 5.10.2, JDK 1.7.0_75, CentOS 6.9.

2017-08-17 19:56:33,582 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Unable to retrieve remote parcel repository manifest
java.util.concurrent.ExecutionException: java.net.ConnectException: Received fatal alert: handshake_failure to https://repository.cask.co/parcels/cdap/4.2/manifest.json
        at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
        at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:104)
        at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:399)
        at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:385)
        at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:352)
        at org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1147)
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1026)
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:664)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:328)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:211)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:372)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:246)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Received fatal alert: handshake_failure to https://repository.cask.co/parcels/cdap/4.2/manifest.json
        at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
        ... 22 more
Caused by: javax.net.ssl.SSLException: Received fatal alert: handshake_failure
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1639)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1607)
        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1776)
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1068)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:890)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:764)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:958)
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:664)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:328)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:211)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:372)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:246)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
        ... 3 more

Thanks,

-Derek

[#2 · Accepted solution]

"Are SSL-enabled custom parcel repositories supported?"

yes, as we can use this https://archive.cloudera.com/cdh5/parcels/

Ths error is likely due to the AWS Cloudfront SNI [0], and the current version of async-http-client-1.7.5.jar that CM uses does not support SNI, see related [1a,b,c]

Testing the URL with openssl excluding the -servername flag

# openssl s_client -connect repository.cask.co:443
CONNECTED(00000003)
140455651178400:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:744:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 247 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

 with -servername [repository|downloads].cask.co

# openssl s_client -connect repository.cask.co:443 -servername repository.cask.co </dev/null | grep "Verify"
depth=4 C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
verify return:1
depth=3 C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = repository.cask.co
verify return:1
DONE
    Verify return code: 0 (ok)

setting-Djavax.net.debug=all in /etc/default/cloudera-scm-server

[...]

export CMF_JAVA_OPTS="... -Djavax.net.debug=all"

[...]

Notice handshake_failure
*** ClientHello, TLSv1
[...]
[Raw read]: length = 2
0000: 02 28                                              .(
New I/O  worker #6, READ: TLSv1 Alert, length = 2
New I/O  worker #6, RECV TLSv1 ALERT:  fatal, handshake_failure
New I/O  worker #6, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Received fatal alert: handshake_failure
New I/O  worker #6, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Received fatal alert: handshake_failure
New I/O  worker #6, called closeOutbound()
New I/O  worker #6, closeOutboundInternal()
New I/O  worker #6, SEND TLSv1 ALERT:  warning, description = close_notify
New I/O  worker #6, WRITE: TLSv1 Alert, length = 2
New I/O  worker #6, called closeInbound()
New I/O  worker #6, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
[Raw write]: length = 7
0000: 15 03 01 00 02 01 00                               .......
New I/O  worker #6, called closeOutbound()
New I/O  worker #6, closeOutboundInternal()

We currently track this internally in /OPSAPS-30976/

Regards,

Michalis

[0] https://aws.amazon.com/about-aws/whats-new/2014/03/05/amazon-cloudront-announces-sni-custom-ssl/

[1a] https://groups.google.com/forum/#!topic/play-framework/T7ZhclgAAMU

[1b] https://github.com/loopj/android-async-http/issues/224

[1c] https://bz.apache.org/bugzilla/show_bug.cgi?id=57935

[#3]

Thanks for the detailed response!