Support Questions
Find answers, ask questions, and share your expertise

Ambari fails to fetch JMX metrics from HBase after enabling SSL

I have an HDP 2.4 cluster with Ambari 2.2.1.1 for which I enabled SSL encryption for Hadoop and HBase components following this documentation and by setting hbase.ssl.enable = true. I've also setup Ambari truststore. Now I get the below error message in HBase Master logs every few seconds.

2016-04-25 17:48:52,704 WARN  [993561082@qtp-1172143954-0] mortbay.log: EXCEPTION
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
        at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:671)
        at sun.security.ssl.InputRecord.read(InputRecord.java:504)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1344)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
        at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

I used `strace` to track it down to Ambari server making HTTP requests to /jmx. On the HBase server, `strace` of the HBase Master PID shows:

[pid  1033] recvfrom(551, "jmx HTTP/1.1\r\nUser-Agent: Java/1.7.0_91\r\nHost: node2.example.com:16010\r\nAccept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\nConnection: keep-alive\r\n\r\n", 161, 0, NULL, NULL) = 161
[pid  1033] sendto(551, "\25\3\1\0\2\2\n", 7, 0, NULL, 0) = 7
[pid  1033] dup2(498, 551)              = 551
[pid  1033] close(551)                  = 0
[pid  1033] write(492, "2016-04-25 15:27:56,675 WARN [1500518225@qtp-1452089300-4] mortbay.log: EXCEPTION \njavax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?\n\tat sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:671)\n

On Ambari server an `strace` of the AmbariServer PID shows:

sendto(915, "GET /jmx HTTP/1.1\r\nUser-Agent: Java/1.7.0_91\r\nHost: node2.example.com:16010\r\nAccept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\nConnection: keep-alive\r\n\r\n", 166, 0, NULL, 0) = 166
poll([{fd=915, events=POLLIN|POLLERR}], 1, 10000) = 1 ([{fd=915, revents=POLLIN}])
recvfrom(915, "\25\3\1\0\2\2\n", 8192, 0, NULL, NULL) = 7

What setting should I change to get Ambari to use SSL to fetch JMX messages?

5 REPLIES 5

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Can you share your ranger-hbase-policymgr-ssl file?

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Expert Contributor

1. This property is actually deprecated. https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml

Have you set: dfs.http.policy?

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Yes, the cluster has dfs.http.policy=HTTPS_ONLY

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Expert Contributor

@Vladimir Zlatkin: Could you solve this issue? I am facing the same problem here.

@Alejandro Fernandez, @swagle: Can you provide more insights on this?