Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari fails to fetch JMX metrics from HBase after enabling SSL

Highlighted

Ambari fails to fetch JMX metrics from HBase after enabling SSL

I have an HDP 2.4 cluster with Ambari 2.2.1.1 for which I enabled SSL encryption for Hadoop and HBase components following this documentation and by setting hbase.ssl.enable = true. I've also setup Ambari truststore. Now I get the below error message in HBase Master logs every few seconds.

2016-04-25 17:48:52,704 WARN  [993561082@qtp-1172143954-0] mortbay.log: EXCEPTION
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
        at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:671)
        at sun.security.ssl.InputRecord.read(InputRecord.java:504)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1344)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
        at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

I used `strace` to track it down to Ambari server making HTTP requests to /jmx. On the HBase server, `strace` of the HBase Master PID shows:

[pid  1033] recvfrom(551, "jmx HTTP/1.1\r\nUser-Agent: Java/1.7.0_91\r\nHost: node2.example.com:16010\r\nAccept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\nConnection: keep-alive\r\n\r\n", 161, 0, NULL, NULL) = 161
[pid  1033] sendto(551, "\25\3\1\0\2\2\n", 7, 0, NULL, 0) = 7
[pid  1033] dup2(498, 551)              = 551
[pid  1033] close(551)                  = 0
[pid  1033] write(492, "2016-04-25 15:27:56,675 WARN [1500518225@qtp-1452089300-4] mortbay.log: EXCEPTION \njavax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?\n\tat sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:671)\n

On Ambari server an `strace` of the AmbariServer PID shows:

sendto(915, "GET /jmx HTTP/1.1\r\nUser-Agent: Java/1.7.0_91\r\nHost: node2.example.com:16010\r\nAccept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\nConnection: keep-alive\r\n\r\n", 166, 0, NULL, 0) = 166
poll([{fd=915, events=POLLIN|POLLERR}], 1, 10000) = 1 ([{fd=915, revents=POLLIN}])
recvfrom(915, "\25\3\1\0\2\2\n", 8192, 0, NULL, NULL) = 7

What setting should I change to get Ambari to use SSL to fetch JMX messages?

5 REPLIES 5

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Can you share your ranger-hbase-policymgr-ssl file?

Highlighted

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Highlighted

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Expert Contributor

1. This property is actually deprecated. https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml

Have you set: dfs.http.policy?

Highlighted

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Yes, the cluster has dfs.http.policy=HTTPS_ONLY
Highlighted

Re: Ambari fails to fetch JMX metrics from HBase after enabling SSL

Expert Contributor

@Vladimir Zlatkin: Could you solve this issue? I am facing the same problem here.

@Alejandro Fernandez, @swagle: Can you provide more insights on this?

Don't have an account?
Coming from Hortonworks? Activate your account here