question Re: Disable/remove auto TLS certificates and create self signed certificate in Support Questions

Disable/remove auto TLS certificates and create self signed certificate

vaibhavm — Tue, 08 Sep 2020 09:45:04 GMT

How to disable/remove auto TLS certificates and create self-signed certificate in Cloudera version 6.2. The Cloudera version I am using is 6.2 having self-signed certificates that expired a few days back. now the Cloudera manager is not restarting. I want to remove the existing ones and create a new self-signed certificate and apply it to the cluster. can anyone help?

Re: Disable/remove auto TLS certificates and create self signed certificate

Mike in Austin — Wed, 09 Sep 2020 14:44:46 GMT

According to this link, you should renew and replace the certificates and points to the procedure just above the link.

Re: Disable/remove auto TLS certificates and create self signed certificate

vaibhavm — Thu, 10 Sep 2020 08:01:38 GMT

@Mike in Austin Thanks for the reply. how can I completely disable the SSL in the cluster? SSL was configured with auto-tls enabled

Re: Disable/remove auto TLS certificates and create self signed certificate

GangWar — Thu, 10 Sep 2020 08:32:58 GMT

@vaibhavm You need two steps process.

1. Disable TLS for CM so that you can access Web UI, for this follow below instruction.

Disable TLS for the CM:

1. Determine Cloudera Manager Database

cat /etc/cloudera-scm-server/db.properties

2. Make database backup

3. get inside the DB. 
#mysql --user=cm --password=cm
#mysql> show databases;
#mysql> use cm;

4. Show TLS related rows 
select * from CONFIGS where attr like '%tls%';

5. Update TLS for web_tls 
update CONFIGS set value = 'false' where attr = 'web_tls';

6. Update TLS for agent_tls 
update CONFIGS set value = 'false' where attr = 'agent_tls';

7. Show TLS related rows 
select * from CONFIGS where attr like '%tls%';

8. Restart Cloudera Manager server process
service cloudera-scm-server restart

2. At this stage you will be able to successfully login into CM Web UI. Now you can disable Auto-TLS (If already enabled) using below method.

--remove the line in /etc/default/cloudera-scm-server that loads cm_init.txt on startup
--then you can turn off TLS in the web UI and remove the TLS configs from the agent config.ini

3. Then you can follow the doc which @Mike in Austin refereed in his comment to enable TLS again from fresh.

Re: Disable/remove auto TLS certificates and create self signed certificate

vaibhavm — Thu, 10 Sep 2020 10:23:53 GMT

@GangWardone what you said. but now I am not able to login with the default username password admin/admin. can you please help with this?

Re: Disable/remove auto TLS certificates and create self signed certificate

GangWar — Thu, 10 Sep 2020 22:02:45 GMT

@vaibhavm Log says it's successful. Try with another browser else you might need to reset the password for local user. Also worth to see if you are using LDAP with CM.

Re: Disable/remove auto TLS certificates and create self signed certificate

vaibhavm — Fri, 11 Sep 2020 11:05:28 GMT

@GangWarI am getting this issue while configuring the SSL for Cloudera manager. can you please help me out with this

2020-09-11 07:03:43,787 ERROR MainThread:com.cloudera.server.cmf.config.components.BeanConfiguration: SSL init failure
java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at org.eclipse.jetty.util.ssl.SslContextFactory.loadTrustStore(SslContextFactory.java:1158)
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:315)
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:248)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.cloudera.server.cmf.config.components.BeanConfiguration.getSslContextFactory(BeanConfiguration.java:126)
at com.cloudera.server.cmf.config.components.BeanConfiguration$$EnhancerBySpringCGLIB$$dea6ffb3.CGLIB$getSslContextFactory$3(<generated>)
at com.cloudera.server.cmf.config.components.BeanConfiguration$$EnhancerBySpringCGLIB$$dea6ffb3$$FastClassBySpringCGLIB$$15a496c1.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
at com.cloudera.server.cmf.config.components.BeanConfiguration$$EnhancerBySpringCGLIB$$dea6ffb3.getSslContextFactory(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1178)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1072)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1136)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1064)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:467)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1178)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1072)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1136)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1064)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:396)
at com.cloudera.server.cmf.Main.findBeanFactory(Main.java:481)
at com.cloudera.server.cmf.Main.findRootApplicationContext(Main.java:476)
at com.cloudera.server.cmf.Main.<init>(Main.java:285)
at com.cloudera.server.cmf.Main.main(Main.java:233)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
... 83 more
2020-09-11 07:03:43,883 INFO MainThread:com.cloudera.server.cmf.Main: Server locale set to: en
2020-09-11 07:03:43,894 INFO MainThread:com.cloudera.server.cmf.components.EmbeddedDbManager: CM server is NOT using embedded postgresql database for itself.
2020-09-11 07:03:43,901 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Adding command schedule listener for global-collect-host-statistics
2020-09-11 07:03:43,905 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Initializing schedule for global-collect-host-statistics
2020-09-11 07:03:43,907 INFO MainThread:com.cloudera.cmf.command.datacollection.ClusterStatsCommandScheduler: No license - using free bundle settings
2020-09-11 07:03:43,911 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Replacing existing schedule for global-collect-host-statistics
2020-09-11 07:03:43,958 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Removed schedule DbCommandSchedule{id=71, commandName=global-collect-host-statistics, displayName=null, description=null}
2020-09-11 07:03:43,960 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Adding schedule - DbCommandSchedule{id=72, commandName=global-collect-host-statistics, displayName=null, description=null}
2020-09-11 07:03:43,983 INFO MainThread:com.cloudera.cmf.scheduler.CmfScheduler: Added command schedule '72' to the scheduler
2020-09-11 07:03:44,047 INFO MainThread:com.cloudera.server.cmf.Main: Starting Auto Upgrade
2020-09-11 07:03:44,050 INFO MainThread:com.cloudera.cmf.service.upgrade.AutoUpgradeHandlerRegistry: No Auto Upgrades required.
2020-09-11 07:03:44,050 INFO MainThread:com.cloudera.server.cmf.Main: Successfully completed Auto Upgrade
2020-09-11 07:03:44,163 INFO MainThread:com.cloudera.server.cmf.Main: Agent RPC connections will use port: 7182
2020-09-11 07:03:44,163 INFO MainThread:com.cloudera.server.cmf.Main: Agent TLS certificates will be validated.
2020-09-11 07:03:44,185 INFO MainThread:com.cloudera.server.common.HttpConnectorServer: Max heartbeat processing thread: 25 and Max threads for CM agent avro http connector: 200
2020-09-11 07:03:44,307 INFO MainThread:com.cloudera.server.common.HttpConnectorServer: HttpConnectorServer port=7182
2020-09-11 07:03:44,307 INFO MainThread:com.cloudera.server.common.HttpConnectorServer: HttpConnectorServer IdleTime=300000
2020-09-11 07:03:44,343 INFO MainThread:org.eclipse.jetty.server.Server: jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_181-b13
2020-09-11 07:03:44,375 WARN MainThread:org.eclipse.jetty.security.SecurityHandler: ServletContext@o.e.j.s.ServletContextHandler@b51e1e1{/,null,STARTING} has uncovered http methods for path: /*
2020-09-11 07:03:44,385 INFO MainThread:org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@b51e1e1{/,null,AVAILABLE}
2020-09-11 07:03:44,394 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2020-09-11 07:03:44,394 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect
at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:224)
at com.cloudera.server.cmf.Main.startAgentServer(Main.java:554)
at com.cloudera.server.cmf.Main.run(Main.java:606)
at com.cloudera.server.cmf.Main.main(Main.java:234)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at org.eclipse.jetty.util.ssl.SslContextFactory.loadTrustStore(SslContextFactory.java:1158)
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:315)
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:248)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:94)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:282)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:394)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:222)
... 3 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
... 25 more
2020-09-11 07:03:53,644 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfull

Re: Disable/remove auto TLS certificates and create self signed certificate

vaibhavm — Fri, 11 Sep 2020 13:39:58 GMT

@GangWar @Mike in Austin I am generating a self signed certificate for my cluster. is it required to generate a certificate for each host(7 hosts in my cluster) or just for the Cloudera host which will be my name node?

Re: Disable/remove auto TLS certificates and create self signed certificate

vaibhavm — Sat, 12 Sep 2020 10:29:52 GMT

@GangWar @Mike in Austin What should be kept in the agentkey.pw while creating the password file?

Re: Disable/remove auto TLS certificates and create self signed certificate

lvazquez — Tue, 21 Jun 2022 00:32:43 GMT

The solution above has been the accepted answer for a long time (since CM 6.x), but the procedure is not complete, at least for Cloudera Manager 7.6 with CDP Base 7.1.6 or above.
After using Auto-TLS and needing to disable it because I needed to rename some nodes, I changed the web_tls and agent_tls settings in the Database, and changed the config.ini to disable TLS in all the agents' config and restarted all the cloudera-scm-* daemons (server/agents) and I was able to log to the UI and reach the hosts, but in "Management >> Security" the Cloudera Manager UI keeps showing "Auto-TLS is Enabled" in the menu bar, and if I try to add a new host it will fail with an error like "Unable to deploy host certificates" ... so it seems CM still has Auto-TLS enable somewhere/somehow.

I have made a dump and double checked there isn't any TLS config parameter enabled in the DB, but there should be some other place from where CM assumes this is enabled.

Re: Disable/remove auto TLS certificates and create self signed certificate

BdE-Big — Fri, 08 Jul 2022 11:59:53 GMT

Remove all TLS related config in CM UI. CM > Settings search for TLS and uncheck everything and delete jks files and password configs

Re: Disable/remove auto TLS certificates and create self signed certificate

lvazquez — Fri, 08 Jul 2022 15:44:57 GMT

I thought it was something like this, but it was hard to believe!

After 2 full installation of CDP base, it seems clear that CDP may have been a big step for the final user, but still has a lot of room for improvement in the sysadmin and devops side of the platform, specially in the way-back or recovery of many central configuration changes (kerberos, TLS) where it really sucks, even when compared with the now ancient HDP3.

Re: Disable/remove auto TLS certificates and create self signed certificate

ClouderaUser123 — Tue, 18 Jul 2023 13:42:02 GMT

@GangWar

This is still a problem in CDP 7.1.8 where there is no possibility of turning off the "Auto-TLS is Enabled" satus in Admin --> Security. Has anyone found the solution? I've now combed through UI settings, db and local files for anything to do with TLS and removed most if it.

I know its turned off but as long as CDP thinks that Auto-TLS is ON I can't run the Auto-TLS setup Wizzard.