Support Questions
Find answers, ask questions, and share your expertise

issue with cloudera management services after configuring TLS

Contributor

Hi everybody

I just try to configure TLS level 1. after I restart the cloudera-scm-server i have this error and i can't have access to the manager web interface.

2017-07-18 15:02:32,325 WARN MainThread:org.mortbay.log: failed Server@4672853b: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (Aucun fichier ou dossier de ce type)
2017-07-18 15:02:32,326 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2017-07-18 15:02:32,333 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (Aucun fichier ou dossier de ce type)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
        at com.cloudera.server.cmf.Main.startAgentServer(Main.java:572)
        at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
        at com.cloudera.server.cmf.Main.run(Main.java:620)
        at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (Aucun fichier ou dossier de ce type)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:639)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
        ... 4 more

thi is the tuto i use: https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cm_sg_config_tls_encr.html#topic_2

 

how can i resove it?

1 ACCEPTED SOLUTION

Contributor

all issues are resolve

i forgot to copy the truststore on the other machines of the cluster

thanks for your help

View solution in original post

18 REPLIES 18

Champion

Did you read and complete Step 0?

 

https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cm_sg_tls_browser.html#xd_583c10bfdb...

 

This will have you create or obtain a server certificate and put it in a Java keystore.  If yes, is it located in the path listed in the exception, /var/lib/cloudera-scm-server/.keystore?  If yes, is it owned by the user that is trying to launch the cloudera-scm-server process, should be cloudera-scm?

Contributor

thanks for your response. i perform level 0 using selfsigned certificate 

https://www.cloudera.com/documentation/enterprise/5-11-x/topics/sg_self_signed_tls.html#sg_self_sign...

my keystore is in /opt/cloudera/security/jks.

can i simply move it on /var/lib/cloudera-scm-server/.keystore?

 

Champion
Yes. Move it and ensure that the user running the cloudera-scm-server has read access to it.

Contributor

thanks for your reply

after moving the keystore into .keystore i have a new error when i restart the manager

2017-07-19 14:46:50,695 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2017-07-19 14:46:50,695 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (est un dossier)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
        at com.cloudera.server.cmf.Main.startAgentServer(Main.java:572)
        at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
        at com.cloudera.server.cmf.Main.run(Main.java:620)
        at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.io.FileNotFoundException: /var/lib/cloudera-scm-server/.keystore (est un dossier)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:639)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
        ... 4 more

Contributor

i think i'm doing a confusion. 

do you have a procedure to help me?

Champion
I do not beyond the Cloudera docs. I have not seen the second error. The Google translation I got was FileNotFoundException (is a record).

To validate that the keystore is good can you run the below command.

keytool -v -list -keystore /var/lib/cloudera-scm-server/.keystore

Contributor

when i enter the command:

Type Keystore : JKS
Fournisseur Keystore : SUN

Votre Keystore contient 1 entrée(s)

Nom d'alias : cmhost
Date de création : 17 juil. 2017
Type d'entrée : PrivateKeyEntry
Longueur de chaîne du certificat : 1

after i have restarted the service i have this error:

 

2017-07-20 09:03:24,082 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2017-07-20 09:03:24,083 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.security.UnrecoverableKeyException: Password must not be null
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
        at com.cloudera.server.cmf.Main.startAgentServer(Main.java:572)
        at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
        at com.cloudera.server.cmf.Main.run(Main.java:620)
        at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.security.UnrecoverableKeyException: Password must not be null
        at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:124)
        at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:55)
        at java.security.KeyStore.getKey(KeyStore.java:792)
        at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:131)
        at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:68)
        at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:259)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:651)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
        ... 4 more

Contributor

is there a way to cancel all the configuration i have done in cloudera manager? i want to restart the configuration since level 0

Champion

Aww I can work with password must not be null.  I assume that the keytool command did not prompt you for a password.  This means that the Java keystore and possible the private key are not password protected.  Most service require that a password be set.  The challenge here is whether you specified a password in the Cloudera Manager configs.  If yes, and you recall it, you can recreate the key and cert in the JKS with that password and bring CM up.

 

Note: the key and JKS password must be the same, CM assumes they are.

 

To revert, you will need to log into the CM database and manually modify it.  Let me track down those instructions.

Champion

Contributor

thanks but But I no longer needed to use rollback

Contributor

IT WORKS 

i have found my error. The folder were i create the truststore (copy of /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/security/cacerts) must be /var/lib/cloudera-scm-server/. i have done all the selfsigned certificate in this folder (/var/lib/cloudera-scm-server/) and it work.

Contributor

now i have acces to Manager

but there is a new message 

WARN 515969315@agentServer-0:org.mortbay.log: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

i have 2 questions: 

normally in which folder trustore must be?

how many trustore must i have?

Champion
That warning indicates that something is talking to CM without using SSL. Did you change all of the agent config files to use_tls=1?

As for the truststore questions. First there is a keystore and a truststore. The keystore stores the key and certificate for a service. This is sensitive as it is the source of how a service identifies itself to another. The truststore just hold the signing certificate and is used by clients to trust any certs signed by the certs in it.

The path /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/security/cacerts looks similar to the location that you would store a system-wide truststore. I think that location is right and the name would be jssecacert or something similar. This means that all Java based program will use this by default without needing to tell the app or client of its location.

Now you don't have to use it; you can create and use your own. And you can have as many as you want although each app, service, client can usually only be configured to use one at a time. Plus, since it is only storing the CA cert why not just have them all in one store to cut down the work.

Note: with self-sign certs, the cert itself become the certificate signing or CA cert and must be put in the truststore.

Contributor

ok I think I understand

in my case; there are two types of encryption: the first type is for HTTPS and the second is for encryption between agents and server. it means that i must have two keystores? 

if yes how can i send public key from the other client to the truststore?

Contributor
2017-07-24 13:22:12,767 WARN 815257673@scm-web-162:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
org.apache.avro.AvroRemoteException: java.net.ConnectException: Connexion refusée
        at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
        at com.sun.proxy.$Proxy111.getAvroHealthReports(Unknown Source)
        at com.cloudera.cmf.protocol.firehose.nozzle.TimeoutNozzleIPC.getAvroHealthReports(TimeoutNozzleIPC.java:127)
        at com.cloudera.cmon.NozzleIPCWrapper.getHealthReports(NozzleIPCWrapper.java:599)
        at com.cloudera.server.web.cmf.HealthReportHelper$GetHealthReportCallable.call(HealthReportHelper.java:502)
        at com.cloudera.server.web.cmf.HealthReportHelper.getHealthReport(HealthReportHelper.java:393)
        at com.cloudera.server.web.cmf.HealthCheckController.hostStatusHealthCheckJSON(HealthCheckController.java:427)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
        at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.net.ConnectException: Connexion refusée
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
        at sun.net.www.http.HttpClient.New(HttpClient.java:308)
        at sun.net.www.http.HttpClient.New(HttpClient.java:326)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
        at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
        at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
        at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
        at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
        at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
        at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
        ... 77 more

i think this is the cause of this issu.

Contributor

all issues are resolve

i forgot to copy the truststore on the other machines of the cluster

thanks for your help

Explorer
Below issue can happen if certifcate is expired? I see in some logs that certificates are expired. Please send documentation for certification renewal.

2019-02-13 23:31:58,038 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
2019-02-13 23:31:58,703 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
2019-02-13 23:32:01,494 INFO 1645307921@scm-web-99151:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 192.168.10.51
2019-02-13 23:32:03,490 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired
; ;