question Re: issue with cloudera management services after configuring TLS in Archives of Support Questions (Read Only)

issue with cloudera management services after configuring TLS

securehadoop — Tue, 18 Jul 2017 13:17:45 GMT

Hi everybody

I just try to configure TLS level 1. after I restart the cloudera-scm-server i have this error and i can't have access to the manager web interface.

2017-07-18 15:02:32,325 WARN MainThread:org.mortbay.log: failed Server@4672853b: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (Aucun fichier ou dossier de ce type)
2017-07-18 15:02:32,326 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2017-07-18 15:02:32,333 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (Aucun fichier ou dossier de ce type)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
        at com.cloudera.server.cmf.Main.startAgentServer(Main.java:572)
        at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
        at com.cloudera.server.cmf.Main.run(Main.java:620)
        at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (Aucun fichier ou dossier de ce type)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:639)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
        ... 4 more

thi is the tuto i use: https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cm_sg_config_tls_encr.html#topic_2

how can i resove it?

Re: issue with cloudera management services after configuring TLS

mbigelow — Tue, 18 Jul 2017 16:21:53 GMT

Did you read and complete Step 0?

https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cm_sg_tls_browser.html#xd_583c10bfdbd326ba-7dae4aa6-147c30d0933--7a61

This will have you create or obtain a server certificate and put it in a Java keystore. If yes, is it located in the path listed in the exception, /var/lib/cloudera-scm-server/.keystore? If yes, is it owned by the user that is trying to launch the cloudera-scm-server process, should be cloudera-scm?

Re: issue with cloudera management services after configuring TLS

securehadoop — Wed, 19 Jul 2017 07:25:16 GMT

thanks for your response. i perform level 0 using selfsigned certificate

https://www.cloudera.com/documentation/enterprise/5-11-x/topics/sg_self_signed_tls.html#sg_self_signed_tls

my keystore is in /opt/cloudera/security/jks.

can i simply move it on /var/lib/cloudera-scm-server/.keystore?

Re: issue with cloudera management services after configuring TLS

mbigelow — Wed, 19 Jul 2017 12:33:00 GMT

Yes. Move it and ensure that the user running the cloudera-scm-server has read access to it.

Re: issue with cloudera management services after configuring TLS

securehadoop — Wed, 19 Jul 2017 12:48:56 GMT

thanks for your reply

after moving the keystore into .keystore i have a new error when i restart the manager

2017-07-19 14:46:50,695 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2017-07-19 14:46:50,695 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (est un dossier)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
        at com.cloudera.server.cmf.Main.startAgentServer(Main.java:572)
        at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
        at com.cloudera.server.cmf.Main.run(Main.java:620)
        at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (est un dossier)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:639)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
        ... 4 more

Re: issue with cloudera management services after configuring TLS

securehadoop — Wed, 19 Jul 2017 14:52:14 GMT

i think i'm doing a confusion.

do you have a procedure to help me?

Re: issue with cloudera management services after configuring TLS

mbigelow — Wed, 19 Jul 2017 14:58:41 GMT

I do not beyond the Cloudera docs. I have not seen the second error. The Google translation I got was FileNotFoundException (is a record).

To validate that the keystore is good can you run the below command.

keytool -v -list -keystore /var/lib/cloudera-scm-server/.keystore

Re: issue with cloudera management services after configuring TLS

securehadoop — Thu, 20 Jul 2017 07:13:26 GMT

when i enter the command:

Type Keystore : JKS
Fournisseur Keystore : SUN

Votre Keystore contient 1 entrée(s)

Nom d'alias : cmhost
Date de création : 17 juil. 2017
Type d'entrée : PrivateKeyEntry
Longueur de chaîne du certificat : 1

after i have restarted the service i have this error:

2017-07-20 09:03:24,082 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2017-07-20 09:03:24,083 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.security.UnrecoverableKeyException: Password must not be null
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
        at com.cloudera.server.cmf.Main.startAgentServer(Main.java:572)
        at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
        at com.cloudera.server.cmf.Main.run(Main.java:620)
        at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.security.UnrecoverableKeyException: Password must not be null
        at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:124)
        at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:55)
        at java.security.KeyStore.getKey(KeyStore.java:792)
        at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:131)
        at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:68)
        at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:259)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:651)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
        ... 4 more

Re: issue with cloudera management services after configuring TLS

securehadoop — Thu, 20 Jul 2017 09:36:42 GMT

is there a way to cancel all the configuration i have done in cloudera manager? i want to restart the configuration since level 0

Re: issue with cloudera management services after configuring TLS

mbigelow — Thu, 20 Jul 2017 15:18:07 GMT

Aww I can work with password must not be null. I assume that the keytool command did not prompt you for a password. This means that the Java keystore and possible the private key are not password protected. Most service require that a password be set. The challenge here is whether you specified a password in the Cloudera Manager configs. If yes, and you recall it, you can recreate the key and cert in the JKS with that password and bring CM up.

Note: the key and JKS password must be the same, CM assumes they are.

To revert, you will need to log into the CM database and manually modify it. Let me track down those instructions.

Re: issue with cloudera management services after configuring TLS

mbigelow — Thu, 20 Jul 2017 15:21:15 GMT

https://community.cloudera.com/t5/Cloudera-Manager-Installation/how-to-rollback-cloudera-manager-tls-configuration-without-UI/m-p/46484/highlight/true#M8455

Re: issue with cloudera management services after configuring TLS

securehadoop — Fri, 21 Jul 2017 09:24:54 GMT

IT WORKS

i have found my error. The folder were i create the truststore (copy of /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/security/cacerts) must be /var/lib/cloudera-scm-server/. i have done all the selfsigned certificate in this folder (/var/lib/cloudera-scm-server/) and it work.

Re: issue with cloudera management services after configuring TLS

securehadoop — Fri, 21 Jul 2017 09:50:43 GMT

now i have acces to Manager

but there is a new message

WARN 515969315@agentServer-0:org.mortbay.log: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

i have 2 questions:

normally in which folder trustore must be?

how many trustore must i have?

Re: issue with cloudera management services after configuring TLS

securehadoop — Fri, 21 Jul 2017 13:27:02 GMT

thanks but But I no longer needed to use rollback

Re: issue with cloudera management services after configuring TLS

mbigelow — Fri, 21 Jul 2017 21:52:31 GMT

That warning indicates that something is talking to CM without using SSL. Did you change all of the agent config files to use_tls=1?

As for the truststore questions. First there is a keystore and a truststore. The keystore stores the key and certificate for a service. This is sensitive as it is the source of how a service identifies itself to another. The truststore just hold the signing certificate and is used by clients to trust any certs signed by the certs in it.

The path /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/security/cacerts looks similar to the location that you would store a system-wide truststore. I think that location is right and the name would be jssecacert or something similar. This means that all Java based program will use this by default without needing to tell the app or client of its location.

Now you don't have to use it; you can create and use your own. And you can have as many as you want although each app, service, client can usually only be configured to use one at a time. Plus, since it is only storing the CA cert why not just have them all in one store to cut down the work.

Note: with self-sign certs, the cert itself become the certificate signing or CA cert and must be put in the truststore.

Re: issue with cloudera management services after configuring TLS

securehadoop — Mon, 24 Jul 2017 10:02:41 GMT

ok I think I understand

in my case; there are two types of encryption: the first type is for HTTPS and the second is for encryption between agents and server. it means that i must have two keystores?

if yes how can i send public key from the other client to the truststore?

Re: issue with cloudera management services after configuring TLS

securehadoop — Mon, 24 Jul 2017 11:27:48 GMT

2017-07-24 13:22:12,767 WARN 815257673@scm-web-162:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

org.apache.avro.AvroRemoteException: java.net.ConnectException: Connexion refusée
        at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
        at com.sun.proxy.$Proxy111.getAvroHealthReports(Unknown Source)
        at com.cloudera.cmf.protocol.firehose.nozzle.TimeoutNozzleIPC.getAvroHealthReports(TimeoutNozzleIPC.java:127)
        at com.cloudera.cmon.NozzleIPCWrapper.getHealthReports(NozzleIPCWrapper.java:599)
        at com.cloudera.server.web.cmf.HealthReportHelper$GetHealthReportCallable.call(HealthReportHelper.java:502)
        at com.cloudera.server.web.cmf.HealthReportHelper.getHealthReport(HealthReportHelper.java:393)
        at com.cloudera.server.web.cmf.HealthCheckController.hostStatusHealthCheckJSON(HealthCheckController.java:427)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
        at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.net.ConnectException: Connexion refusée
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
        at sun.net.www.http.HttpClient.New(HttpClient.java:308)
        at sun.net.www.http.HttpClient.New(HttpClient.java:326)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
        at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
        at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
        at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
        at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
        at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
        at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
        ... 77 more

i think this is the cause of this issu.

Re: issue with cloudera management services after configuring TLS

securehadoop — Wed, 26 Jul 2017 09:38:20 GMT

all issues are resolve

i forgot to copy the truststore on the other machines of the cluster

thanks for your help

Re: issue with cloudera management services after configuring TLS

pra_big — Wed, 13 Feb 2019 14:27:45 GMT

Below issue can happen if certifcate is expired? I see in some logs that certificates are expired. Please send documentation for certification renewal.

2019-02-13 23:31:58,038 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
2019-02-13 23:31:58,703 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
2019-02-13 23:32:01,494 INFO 1645307921@scm-web-99151:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 192.168.10.51
2019-02-13 23:32:03,490 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired