Add host fails with "Failed to generate certificates" message

zinal — Thu, 12 Aug 2021 13:36:28 GMT

Hello experts!

Trying to add the host to an existing SSL-protected demo CDP 7.1.6 cluster (on IBM Power servers, ppc64le).

The "Install Agents" wizard step fails with the error "Installation failed. Failed to copy installation files."

When looking into the "Details" link, the following is printed there:

/tmp/scm_prepare_node.gvqeyBBt
Failed to generate certificates for cldr62.ibmcc.ru

/opt/cloudera/cm-agent/bin/certmanager is present on both the Cloudera Manager host and on the machine to be added.

Can someone please recommend how to resolve this issue?
Cloudera Manager logs do not seem to reveal anything useful:

2021-08-12 16:14:24,928 INFO scm-web-499421:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/add-hosts -wizard/install 2021-08-12 16:14:24,936 INFO scm-web-499421:com.cloudera.server.cmf.node.NodeConfiguratorService: Creating request with id 16 2021-08-12 16:14:24,937 INFO scm-web-499421:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing Global command GlobalHostInstall Glo balHostInstallCommandArgs{sshPort=22, userName=root, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, parallelInstallCount=10, cmRepoUrl=http://cldrmngr/repos/cm7/7.3.1/redhat7ppc/yum, gpgKeyCustomUrl=null, gpgKeyOverrideBundle=<none>, unlimitedJCE=false, javaInstal lStrategy=AUTO, agentUserMode=ROOT, cdhVersion=-1, cdhRelease=NONE, cdhRepoUrl=null, buildCertCommand={{TEMP_DIR}}, sslCertHostname=cldrmng r.ibmcc.ru, reqId=16, skipPackageInstall=false, skipCloudConfig=false, proxyProtocol=HTTP, proxyServer=null, proxyPort=0, proxyUserName=nul l, proxyPassword=REDACTED, cmca=REDACTED, hostCerts=<none>, customTrustStorePath=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_trus tstore.jks, customTrustStorePassword=odp8Z2Mgzdz9RXUPIv9EKIsBkaHZLNN0BHdkEOY77en, customTrustStoreType=jks, hosts=[cldr62.ibmcc.ru], existi ngHosts=[], agentReportedHostnames=null}. 2021-08-12 16:14:24,938 INFO scm-web-499421:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546342398 work: Execute 1 steps in se quence 2021-08-12 16:14:24,938 INFO scm-web-499421:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546342398 work: Install on 1 hosts. 2021-08-12 16:14:24,938 INFO scm-web-499421:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546342398 work: Install on cldr62.ibm cc.ru. 2021-08-12 16:14:24,942 INFO scm-web-499421:com.cloudera.server.cmf.node.NodeConfiguratorService: Adding password-based configurator for cl dr62.ibmcc.ru 2021-08-12 16:14:24,943 INFO scm-web-499421:com.cloudera.server.cmf.node.NodeConfiguratorService: Submitted configurator for cldr62.ibmcc.r u with id 17 2021-08-12 16:14:24,943 INFO NodeConfiguratorThread-16-0:com.cloudera.cmf.model.HostInstallArgs: Deprecated option for unlimited strength J CE. Value set to False. 2021-08-12 16:14:24,943 INFO scm-web-499421:com.cloudera.cmf.service.ServiceHandlerRegistry: Global Command GlobalHostInstall launched with id=1546342398 2021-08-12 16:14:24,991 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Transition ing from INIT (PT0.049S) to CONNECT 2021-08-12 16:14:24,991 INFO NodeConfiguratorThread-16-0:net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0_14_0 2021-08-12 16:14:25,001 INFO NodeConfiguratorThread-16-0:net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4 2021-08-12 16:14:25,008 INFO scm-web-499421:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/install, Status:200 2021-08-12 16:14:25,010 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:1546342398 2021-08-12 16:14:25,022 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Transitioning from CONNECT (PT0.031S) to AUTHENTICATE 2021-08-12 16:14:25,075 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Transitioning from AUTHENTICATE (PT0.053S) to MAKE_TEMP_DIR 2021-08-12 16:14:25,193 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfigurator: Executing mktemp -d /tmp/scm_prepare_node.XXXXXXXX on cldr62.ibmcc.ru 2021-08-12 16:14:25,210 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Transitioning from MAKE_TEMP_DIR (PT0.135S) to COPY_FILES 2021-08-12 16:14:25,267 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfigurator: Using default key bundle URL 2021-08-12 16:14:25,452 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.HostCertConfigurator: Creating temporary directory for certificate generation. 2021-08-12 16:14:25,525 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.HostCertConfigurator: Using host certificate generator command: /opt/cloudera/cm-agent/bin/certmanager --location /tmp/generateHostCerts16944646874431887540 gen_node_cert --output=- 2021-08-12 16:14:26,824 ERROR NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.HostCertConfigurator: Failed to generate certificates for cldr62.ibmcc.ru: 2021-08-12 16:14:26,824 ERROR NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.HostCertConfigurator: Certificate generation failed. Temporary directory is at: /tmp/generateHostCerts16944646874431887540 2021-08-12 16:14:26,824 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Setting COPY_FILES as failed and done state 2021-08-12 16:14:26,824 INFO NodeConfiguratorThread-16-0:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION 2021-08-12 16:14:26,824 INFO NodeConfiguratorThread-16-0:com.cloudera.cmf.model.HostInstallArgs: Deprecated option for unlimited strength JCE. Value set to False. 2021-08-12 16:14:27,774 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully. 2021-08-12 16:14:30,015 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:1546342398 2021-08-12 16:14:30,019 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 1546342398 Failed to complete installation on host cldr62.ibmcc.ru. 2021-08-12 16:14:30,019 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546342398(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation. 2021-08-12 16:14:30,020 INFO CommandPusher-1:com.cloudera.cmf.command.components.CommandStorage: Invoked delete temp files for command:DbCommand{id=1546342398, name=GlobalHostInstall} at dir:/var/lib/cloudera-scm-server/temp/commands/1546342398

Re: Add host fails with "Failed to generate certificates" message

GangWar — Sun, 15 Aug 2021 03:06:50 GMT

@zinal I would request you to check if you have root privileges on the host because that's needed for Auto-TLS. Also check the permission on the directory where CM Auto-TLS stores the certificate.

Lastly you can bypass this issue by copying certs manually from below dir:

Temporary directory is at: /tmp/generateHostCerts16944646874431887540

to the Auto-TLS dirs on the host and modify the config.ini file (refer the config.ini file from working host) and restart the agent.

Re: Add host fails with "Failed to generate certificates" message

cjervis — Thu, 19 Aug 2021 20:44:44 GMT

@zinal Have you resolved your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Re: Add host fails with "Failed to generate certificates" message

mzinal — Fri, 20 Aug 2021 08:51:13 GMT

Hello colleagues!

Thank you for proposing the solution, I believe that it should work - although I still had not a chance to check that.

Our HW people have ruined the test cluster, and I am still waiting for it to be recovered.

Re: Add host fails with "Failed to generate certificates" message

mzinal — Fri, 20 Aug 2021 09:01:44 GMT

By the way, I do not see the "Accept as solution" button - so I cannot do that.

UPDATE: had to re-login in another browser - simple "Logout" did not do the trick.

😞

Re: Add host fails with "Failed to generate certificates" message

cjervis — Fri, 20 Aug 2021 12:23:47 GMT

Thank you for marking the solution. It appears that you have two different community accounts. Of your two accounts, only the one that posted the question can mark the solution. If you have any further questions on the accounts or using the community, feel free to reach out to me via private message.

question Add host fails with "Failed to generate certificates" message in Support Questions

Add host fails with "Failed to generate certificates" message

Re: Add host fails with "Failed to generate certificates" message

Re: Add host fails with "Failed to generate certificates" message

Re: Add host fails with "Failed to generate certificates" message

Re: Add host fails with "Failed to generate certificates" message

Re: Add host fails with "Failed to generate certificates" message