Renew certificates on a CDP cluster that has auto tls enabled with a Root CA

[#1]

Hi
I have an existing CDP 7.1.x cluster with Auto-tls enabled during the creation of the cluster. I followed the use case 2: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/security-encrypting-data-in-transit/topics/cm... to use an existing Root CA. Recently the certificates expired and I'm trying to renew them. I've a couple of questions from the documentation.

1. In the above page, it mentions "In this use case, rotation of the Auto-TLS certificate authority is not supported. Cloudera recommends creating an intermediate CA with a long lifetime. The host certificates can be rotated by using the generateHostCerts API." - Should I use this to generate the host certs. If so, can I get an example of the API call and it's usage.

2. Or should I use this use case 3: https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/security-encrypting-data-in-transit/topics/cm.... Generate the certificates myself and use the generateCmCa api?

I don't mind the using the UI too, but I don't think that's feasible with a different Root CA case.  Can you suggest how can I go about this please?

Thanks

[#2]

@namteen Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our CDP experts @venkatsambath @aakulov @PabitraDas  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Hello @namteen 

Thank you for reaching out yes for renewing

Yes you will need to use Auto-TLS use case 3 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/security-encrypting-data-in-transit/topics/cm...

Also you are correct from UI you can only generate a cloudera manager signed certificate for your custom ca signed certificate you can generate the certificate yourself and pass those in generatecmca API as mentioned in use case 3

[#4]

Hi, I tried following the use case 3. I generated the certificates for all the hosts. When I ran the generateCmCa api, I'm running into this error:

Entering HTTP Operation: Method:POST, Path:/v41/cm/commands/generateCmca
INFO scm-web-77659:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing Global command GenerateCMCACommand GenerateCmcaCmdArgs{sshPort=22, userName=REDACTED, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, customCA=true, interpretAsFilenames=true, additionalArguments=null, location=/opt/cloudera/CMCA}.
INFO scm-web-77659:com.cloudera.cmf.command.GenerateCmcaCommand: {CLUSTER_NAME} has Kerberos enabled and will be reconfigured to use SASL
INFO scm-web-77659:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546436812 work: Execute 14 steps in sequence
INFO scm-web-77659:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546436812 work: Generate a CMCA and enable Auto-TLS.
INFO scm-web-77659:com.cloudera.cmf.command.GenerateCmcaCmdWork: Determined CMCA location: /var/lib/cloudera-scm-server/certmanager
INFO scm-web-77659:com.cloudera.cmf.command.GenerateCmcaCmdWork: Modifying init file if present: /var/lib/cloudera-scm-server/certmanager/cm_init.txt
INFO scm-web-77659:com.cloudera.cmf.command.GenerateCmcaCmdWork: Generating CMCA
INFO scm-web-77659:com.cloudera.cmf.command.CertmanagerRunner: Running CMCA command with args: [setup_custom_certdir, --host-cert, REDACTED, --host-key, REDACTED, --ca-cert, REDACTED, --keystore-pw-file, /tmp/auto-tls/keys/key.pwd, --truststore-pw-file, REDACTED, --configure-services, --skip-cm-init, --override, keystore_type=jks]
ERROR scm-web-77659:com.cloudera.cmf.command.CertmanagerRunner: Failed to run CMCA command, return code: 1, stderr:
INFO:root:certmanager not running as root
INFO:root:Logging to /var/log/cloudera-scm-agent/certmanager.log
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/bin/certmanager", line 11, in <module>
load_entry_point('cmf==7.6.7', 'console_scripts', 'certmanager')()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py", line 2857, in main
return certmanager(obj=argparse.Namespace())
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/click/decorators.py", line 27, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py", line 2694, in setup_custom_certdir
truststore_password)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py", line 2014, in setup_server_with_custom_certs
self.copy_node_cert(None, hostname)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py", line 1798, in copy_node_cert
keystore_file, hostname, password)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py", line 1607, in _write_keystore_file
raise Exception("Failed to generate host pkcs12 file.")
Exception: Failed to generate host pkcs12 file.

WARN scm-web-77659:com.cloudera.cmf.command.flow.CmdStep: Command 1546436812 Unexpected exception during doWork
java.lang.IllegalStateException: Failed to run CMCA command, return code: 1


The ssh user has root permissions assigned. Can you help me with this please @upadhyayk04 

Thank you

[#5]

@venkatsambath @aakulov @PabitraDas  @upadhyayk04 Hi, do you have any insights here? Thanks!

[#6 · Accepted solution]

Hello @namteen ,

The below error indicates there are issues with the certificates or keys. Can you please confirm if all the certificates are signed by the CM valid CA none is expired also the key and host combination that you are providing is correct

File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/tools/cert.py", line 1607, in _write_keystore_file
raise Exception("Failed to generate host pkcs12 file.")
Exception: Failed to generate host pkcs12 file.

You can check /var/log/cloudera-scm-agent/certmanager.log on the CM server host as well to further check the error

>> To check whether the certificate is correct or not run the following command

# openssl x509 -in <cert-file> -noout -text

To check whether cert and key combination is correct or not match output of below commands

#openssl x509 -noout -modulus -in server.pem | openssl md5

#openssl rsa -noout -modulus -in server.key | openssl md5

where server.pem is the certificate and server.key is the key

[#7]

Hi, yes it was a problem with incorrect pass phrase being passed to the keystorePassword.

[#8]

@namteen Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  Thanks.