Disable/remove auto TLS certificates and create self signed certificate

[#1]

How to disable/remove auto TLS certificates and create self-signed certificate in Cloudera version 6.2. The Cloudera version I am using is 6.2 having self-signed certificates that expired a few days back. now the Cloudera manager is not restarting. I want to remove the existing ones and create a new self-signed certificate and apply it to the cluster. can anyone help?

[#2]

According to this link, you should renew and replace the certificates and points to the procedure just above the link.

[#3]

@Mike in Austin Thanks for the reply. how can I completely disable the SSL in the cluster? SSL was configured with auto-tls enabled

[#4 · Accepted solution]

@vaibhavm You need two steps process. 

1. Disable TLS for CM so that you can access Web UI, for this follow below instruction. 

Disable TLS for the CM:

1. Determine Cloudera Manager Database

cat /etc/cloudera-scm-server/db.properties

2. Make database backup

3. get inside the DB. 
#mysql --user=cm --password=cm
#mysql> show databases;
#mysql> use cm;

4. Show TLS related rows 
select * from CONFIGS where attr like '%tls%';

5. Update TLS for web_tls 
update CONFIGS set value = 'false' where attr = 'web_tls';

6. Update TLS for agent_tls 
update CONFIGS set value = 'false' where attr = 'agent_tls';

7. Show TLS related rows 
select * from CONFIGS where attr like '%tls%';

8. Restart Cloudera Manager server process
service cloudera-scm-server restart

2. At this stage you will be able to successfully login into CM Web UI. Now you can disable Auto-TLS (If already enabled) using below method. 

--remove the line in /etc/default/cloudera-scm-server that loads cm_init.txt on startup
--then you can turn off TLS in the web UI and remove the TLS configs from the agent config.ini

 3. Then you can follow the doc which @Mike in Austin refereed in his comment to enable TLS again from fresh. 

[#5]

@GangWardone what you said. but now I am not able to login with the default username password admin/admin. can you please help with this?

[#6]

@vaibhavm Log says it's successful. Try with another browser else you might need to reset the password for local user. Also worth to see if you are using LDAP with CM. 

[#7]

@GangWarI am getting this issue while configuring the SSL for Cloudera manager. can you please help me out with this

2020-09-11 07:03:43,787 ERROR MainThread:com.cloudera.server.cmf.config.components.BeanConfiguration: SSL init failure
java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at org.eclipse.jetty.util.ssl.SslContextFactory.loadTrustStore(SslContextFactory.java:1158)
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:315)
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:248)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.cloudera.server.cmf.config.components.BeanConfiguration.getSslContextFactory(BeanConfiguration.java:126)
at com.cloudera.server.cmf.config.components.BeanConfiguration$$EnhancerBySpringCGLIB$$dea6ffb3.CGLIB$getSslContextFactory$3(<generated>)
at com.cloudera.server.cmf.config.components.BeanConfiguration$$EnhancerBySpringCGLIB$$dea6ffb3$$FastClassBySpringCGLIB$$15a496c1.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
at com.cloudera.server.cmf.config.components.BeanConfiguration$$EnhancerBySpringCGLIB$$dea6ffb3.getSslContextFactory(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1178)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1072)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1136)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1064)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:467)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1178)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1072)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1136)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1064)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:396)
at com.cloudera.server.cmf.Main.findBeanFactory(Main.java:481)
at com.cloudera.server.cmf.Main.findRootApplicationContext(Main.java:476)
at com.cloudera.server.cmf.Main.<init>(Main.java:285)
at com.cloudera.server.cmf.Main.main(Main.java:233)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
... 83 more
2020-09-11 07:03:43,883 INFO MainThread:com.cloudera.server.cmf.Main: Server locale set to: en
2020-09-11 07:03:43,894 INFO MainThread:com.cloudera.server.cmf.components.EmbeddedDbManager: CM server is NOT using embedded postgresql database for itself.
2020-09-11 07:03:43,901 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Adding command schedule listener for global-collect-host-statistics
2020-09-11 07:03:43,905 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Initializing schedule for global-collect-host-statistics
2020-09-11 07:03:43,907 INFO MainThread:com.cloudera.cmf.command.datacollection.ClusterStatsCommandScheduler: No license - using free bundle settings
2020-09-11 07:03:43,911 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Replacing existing schedule for global-collect-host-statistics
2020-09-11 07:03:43,958 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Removed schedule DbCommandSchedule{id=71, commandName=global-collect-host-statistics, displayName=null, description=null}
2020-09-11 07:03:43,960 INFO MainThread:com.cloudera.cmf.service.CommandScheduleListener: Adding schedule - DbCommandSchedule{id=72, commandName=global-collect-host-statistics, displayName=null, description=null}
2020-09-11 07:03:43,983 INFO MainThread:com.cloudera.cmf.scheduler.CmfScheduler: Added command schedule '72' to the scheduler
2020-09-11 07:03:44,047 INFO MainThread:com.cloudera.server.cmf.Main: Starting Auto Upgrade
2020-09-11 07:03:44,050 INFO MainThread:com.cloudera.cmf.service.upgrade.AutoUpgradeHandlerRegistry: No Auto Upgrades required.
2020-09-11 07:03:44,050 INFO MainThread:com.cloudera.server.cmf.Main: Successfully completed Auto Upgrade
2020-09-11 07:03:44,163 INFO MainThread:com.cloudera.server.cmf.Main: Agent RPC connections will use port: 7182
2020-09-11 07:03:44,163 INFO MainThread:com.cloudera.server.cmf.Main: Agent TLS certificates will be validated.
2020-09-11 07:03:44,185 INFO MainThread:com.cloudera.server.common.HttpConnectorServer: Max heartbeat processing thread: 25 and Max threads for CM agent avro http connector: 200
2020-09-11 07:03:44,307 INFO MainThread:com.cloudera.server.common.HttpConnectorServer: HttpConnectorServer port=7182
2020-09-11 07:03:44,307 INFO MainThread:com.cloudera.server.common.HttpConnectorServer: HttpConnectorServer IdleTime=300000
2020-09-11 07:03:44,343 INFO MainThread:org.eclipse.jetty.server.Server: jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_181-b13
2020-09-11 07:03:44,375 WARN MainThread:org.eclipse.jetty.security.SecurityHandler: ServletContext@o.e.j.s.ServletContextHandler@b51e1e1{/,null,STARTING} has uncovered http methods for path: /*
2020-09-11 07:03:44,385 INFO MainThread:org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@b51e1e1{/,null,AVAILABLE}
2020-09-11 07:03:44,394 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2020-09-11 07:03:44,394 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect
at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:224)
at com.cloudera.server.cmf.Main.startAgentServer(Main.java:554)
at com.cloudera.server.cmf.Main.run(Main.java:606)
at com.cloudera.server.cmf.Main.main(Main.java:234)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
at org.eclipse.jetty.util.ssl.SslContextFactory.loadTrustStore(SslContextFactory.java:1158)
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:315)
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:248)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:94)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:282)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:394)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:222)
... 3 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
... 25 more
2020-09-11 07:03:53,644 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfull

[#8]

@GangWar @Mike in Austin I am generating a self signed certificate for my cluster. is it required to generate a certificate for each host(7 hosts in my cluster) or just for the Cloudera host which will be my name node?

[#9]

@GangWar @Mike in Austin What should be kept in the agentkey.pw while creating the password file?

[#10]

The solution above has been the accepted answer for a long time (since CM 6.x), but the procedure is not complete, at least for Cloudera Manager 7.6 with CDP Base 7.1.6 or above.
After using Auto-TLS and needing to disable it because I needed to rename some nodes, I changed the web_tls and agent_tls settings in the Database, and changed the config.ini to disable TLS in all the agents' config and restarted all the cloudera-scm-* daemons (server/agents) and I was able to log to the UI and reach the hosts, but in "Management >> Security" the Cloudera Manager UI keeps showing "Auto-TLS is Enabled" in the menu bar, and if I try to add a new host it will fail with an error like "Unable to deploy host certificates" ... so it seems CM still has Auto-TLS enable somewhere/somehow.

I have made a dump and double checked there isn't any TLS config parameter enabled in the DB, but there should be some other place from where CM assumes this is enabled.