Host Inspector TLS issue

[#1]

We just upgraded our Cloudera Manager from 5.15 to 6.1.0

TLS security was enabled, and it works on server/agent communication and for all the CDH web interfaces. No specific issue there.

We just have problems with the CM "Host Inspector" and "Security Inspector", that cannot run on any host, failing with the message

"IOException thrown while collecting data from host: Unrecognized SSL message, plaintext connection?"

Our CA certificates are included in jssecacerts, we restarted the server/agents, and every other communication seems to work.

Any idea?

Thanks

[#2]

There is a known issue. "Restarting agent does not restart status_server (agent listener)" where the status_server does not know to use TLS until restarted manually as the agent isn't restarting it properly in CM 6.0.x and CM 6.1.0.

Restart status_server by doing the following:
/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

[#3]

It did not work.

There were supervisord processes older that the latest restart.

Running your suggested command on all the clients did not work.

We also completely killed all server/agents, and killed the lingering supervisord processes, before restarting the agents, and nothing changed.

Consider that TLS used to work in 5.15, and still work correctly for everything in the Manager and in the CDH communications, but the Host/Security Inspectors

Is there a way to execute the failing mgmt/mgmt.sh from the console to get some additional insight on the issue?

Thanks anyway.

[#4]

It seems to be a bug,

Internal Cloudera Jira for reference:

CDH-76040

Cloudera is evaluating the issue and determining the best course of action.

Currently, the issue is in CDH 6.1 as well as CDH 6.0x

check out below link:

https://community.cloudera.com/t5/Cloudera-Manager-Installation/HTTP-error-CM-Agent-with-YARN-HA-on-...

[#5]

@mmmunafo,

Restarting the status server will not help since the underlying issue will remain the same.

Also, CDH-76040 is not the cause of this issue as it is specific to the Resource Manager.  CDH-76040 is actually resolved in 6.1.1 and 6.2 (both available).

The problem you are seeing is likely caused by a new feature added in Cloudera Manager 6.1 that attempts to secure port 9000 (the status server port that the Cloudera Manager agent uses to respond ot requests from Cloudera Manager).  The agent will detect that it should run the host inspector command, but it when Cloudera Manager attempts to download results on port 9000, it thinks that TLS should be used when port 9000 is not listening via TLS.

The issue impacts CM 6.1 but was fixed in Cloudera Manager 6.2.  If you can upgrade to CM 6.2 (CDH does not need to be upgraded right away) then this issue should go away.

For reference, the fix is associated with internal Cloudera Jira OPSAPS-48958.

Note that the issue you are facing occurs when agent communication is encrypted, but the agents have no keys, certificates, or truststore configured.  Apart from upgrading to 6.2, the other solution is to configure all agents with their own keys, certificates, and "verify_cert_file" configurations that would normally be there if CM Agent authentication was enabled.

[#6]

Thank you.

We will see if the problem is solved when we will be able to update the system (we are currently in a production phase and we refrain to introduce changes unless strictly necessary).

[#7]

2019-04-13 23:49:25,547 WARN New I/O worker #19:com.cloudera.server.cmf.HeartbeatRequester: Error requesting heartbeat of host id 010a1e08-ccb1-44c3-8346-2940070d5b9f
java.net.ConnectException: https://xxxxxx.bdg112:9000
at com.ning.http.client.providers.netty.request.NettyConnectListener.onFutureFailure(NettyConnectListener.java:133)
at com.ning.http.client.providers.netty.request.NettyConnectListener.access$200(NettyConnectListener.java:37)
at com.ning.http.client.providers.netty.request.NettyConnectListener$1.operationComplete(NettyConnectListener.java:104)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:395)
at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:362)
at org.jboss.netty.handler.ssl.SslHandler.channelDisconnected(SslHandler.java:575)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
at org.jboss.netty.channel.Channels.close(Channels.java:828)
at org.jboss.netty.handler.ssl.SslHandler$ClosingChannelFutureListener.operationComplete(SslHandler.java:1542)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409)
at org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:145)
at org.jboss.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1502)
at org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:514)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:54)
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleDownstream(HttpClientCodec.java:97)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:109)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.jboss.netty.channel.Channels.close(Channels.java:812)
at org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:205)
at com.ning.http.client.providers.netty.channel.Channels.silentlyCloseChannel(Channels.java:48)
at com.ning.http.client.providers.netty.channel.ChannelManager.closeChannel(ChannelManager.java:376)
at com.ning.http.client.providers.netty.handler.Processor.exceptionCaught(Processor.java:184)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:536)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:862)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
... 62 more
2019-04-13 23:49:25,648 INFO scm-web-110:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/v31/clusters/BigData/commands/perfInspector, Status:200
2019-04-13 23:49:25,677 WARN avro-servlet-hb-processor-1:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=72 name=null host=36353fc5-c4e3-4e4a-93b3-f4883ccda7d0/xxxxxx.bdg110
2019-04-13 23:49:25,685 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=69 name=null host=e123aacf-fc47-43d8-a285-9c03cad95e12/xxxxxx.bdg113
2019-04-13 23:49:25,712 WARN avro-servlet-hb-processor-1:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=70 name=null host=da8b59cc-68ca-4cd9-97c7-f62ee1db44e4/xxxxxx.bdg111
2019-04-13 23:49:25,993 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=71 name=null host=010a1e08-ccb1-44c3-8346-2940070d5b9f/xxxxxx.bdg112
2019-04-13 23:49:34,546 WARN AgentResultFetcher-0:com.cloudera.cmf.service.AgentResultFetcher: Error fetching result from agent.
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.jav...)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at com.cloudera.server.common.ConnectionUtils.readAgentUrlWithTimeouts(ConnectionUtils.java:75)
at com.cloudera.server.common.ConnectionUtils.readAgentUrlWithTimeouts(ConnectionUtils.java:61)
at com.cloudera.cmf.service.AgentResultFetcher.read(AgentResultFetcher.java:172)
at com.cloudera.cmf.service.AgentResultFetcher$ResultFetcher.run(AgentResultFetcher.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-04-13 23:49:34,638 WARN CommandPusher:com.cloudera.cmf.command.flow.SingleRequestAgentResultFetcherWorkOutput: Unable to fetch result from agent.
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.jav...)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at com.cloudera.server.common.ConnectionUtils.readAgentUrlWithTimeouts(ConnectionUtils.java:75)
at com.cloudera.server.common.ConnectionUtils.readAgentUrlWithTimeouts(ConnectionUtils.java:61)
at com.cloudera.cmf.service.AgentResultFetcher.read(AgentResultFetcher.java:172)
at com.cloudera.cmf.service.AgentResultFetcher$ResultFetcher.run(AgentResultFetcher.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[#8]

@wusj,

It does indeed look like the same issue, but you stated you installed CDH 6.2.  I assume then you have also installed Cloudera Manager 6.2 packages on *both* agents and Cloudera Manager.  The fix is in the agents code, so you need to make sure all agents are version 6.2 and that they have been restarted after upgrading the packages.

I just tested by upgrading from Cloudera Manager 6.1 to 6.2 and the issue was resolved in my environment without any configuration changes.

If you did restart CM and the agents after installing CM 6.2, then please share the following from one of the hosts having the problem:

grep -v -e '^[[:space:]]*$' -e '^#' /etc/cloudera-scm-agent/config.ini

This will show us your agent configuration.  Feel free to not include your CM hostname as we are mainly concerned with your [security] section.

Ben

[#9]

@wusj,

I was a bit unclear in my previous update regarding the packages to install.  The CM 6.2 Agent package needs to be installed on each of your cluster hosts and the agent needs to be restarted with "service cloudera-scm-agent restart" or "systemctl restart cloudera-scm-agent" in order for the code fix to be used.

[#10]

The openssl tool can be run from the host that is running the Cloudera Manager Agent or client service that should be inspected for connectivity issues. You should also test whether the certificate in use by the host is recognized by a trusted CA during the TLS/SSL negotiation.

Use the following command to inspect the connection.

$ openssl s_client -connect [host.fqdn.name]:[port]

For example:

$ openssl s_client -connect test1.sec.cloudera.com:7183

A return code 0 means openssl was able to establish trust of the server through its library of trusted public CAs. If the certificate was self-signed (recommended only on test clusters) or provided by a private CA, it might be necessary to add the private CA or self-signed certificate to the truststore using the opensslcommand. Adding the path to the root CA, -CAfile </path/to/root-ca.pem>, allows openssl to verify your self-signed or private CA-signed certificate as follows:

$ openssl s_client -connect test1.sec.cloudera.com:7183 -CAfile \
/opt/cloudera/security/CAcerts/RootCA.pem

Note that providing only the Root CA certificate is necessary to establish trust for this test. The result from the command is successful when you see the return code 0 as follows:

...
 Verify return code: 0 (ok)
---

By default, the Cloudera Manager Server writes logs to the /etc/cloudera-scm-server/cloudera-scm-server.log file on startup. Successful start of the server process with the certificate will show logs similar to the following:

2014-10-06 21:33:47,515 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.2 
2014-10-06 21:33:47,572 INFO WebServerImpl:org.mortbay.log: Started SslSelectChannelConnector@0.0.0.0:7183 
2014-10-06 21:33:47,573 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180 
2014-10-06 21:33:47,573 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

Uploading Diagnostic Bundles