Add host fails with "Failed to generate certificates" message

[#1]

Hello experts!

Trying to add the host to an existing SSL-protected demo CDP 7.1.6 cluster (on IBM Power servers, ppc64le).

The "Install Agents" wizard step fails with the error "Installation failed. Failed to copy installation files."

When looking into the "Details" link, the following is printed there:

/tmp/scm_prepare_node.gvqeyBBt
Failed to generate certificates for cldr62.ibmcc.ru

/opt/cloudera/cm-agent/bin/certmanager is present on both the Cloudera Manager host and on the machine to be added.

Can someone please recommend how to resolve this issue?
Cloudera Manager logs do not seem to reveal anything useful:

2021-08-12 16:14:24,928 INFO scm-web-499421:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/add-hosts
-wizard/install
2021-08-12 16:14:24,936 INFO scm-web-499421:com.cloudera.server.cmf.node.NodeConfiguratorService: Creating request with id 16
2021-08-12 16:14:24,937 INFO scm-web-499421:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing Global command GlobalHostInstall Glo
balHostInstallCommandArgs{sshPort=22, userName=root, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, parallelInstallCount=10, 
cmRepoUrl=http://cldrmngr/repos/cm7/7.3.1/redhat7ppc/yum, gpgKeyCustomUrl=null, gpgKeyOverrideBundle=<none>, unlimitedJCE=false, javaInstal
lStrategy=AUTO, agentUserMode=ROOT, cdhVersion=-1, cdhRelease=NONE, cdhRepoUrl=null, buildCertCommand={{TEMP_DIR}}, sslCertHostname=cldrmng
r.ibmcc.ru, reqId=16, skipPackageInstall=false, skipCloudConfig=false, proxyProtocol=HTTP, proxyServer=null, proxyPort=0, proxyUserName=nul
l, proxyPassword=REDACTED, cmca=REDACTED, hostCerts=<none>, customTrustStorePath=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_trus
tstore.jks, customTrustStorePassword=odp8Z2Mgzdz9RXUPIv9EKIsBkaHZLNN0BHdkEOY77en, customTrustStoreType=jks, hosts=[cldr62.ibmcc.ru], existi
ngHosts=[], agentReportedHostnames=null}.
2021-08-12 16:14:24,938 INFO scm-web-499421:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546342398 work: Execute 1 steps in se
quence
2021-08-12 16:14:24,938 INFO scm-web-499421:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546342398 work: Install on 1 hosts.
2021-08-12 16:14:24,938 INFO scm-web-499421:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546342398 work: Install on cldr62.ibm
cc.ru.
2021-08-12 16:14:24,942 INFO scm-web-499421:com.cloudera.server.cmf.node.NodeConfiguratorService: Adding password-based configurator for cl
dr62.ibmcc.ru
2021-08-12 16:14:24,943 INFO scm-web-499421:com.cloudera.server.cmf.node.NodeConfiguratorService: Submitted configurator for cldr62.ibmcc.r
u with id 17
2021-08-12 16:14:24,943 INFO NodeConfiguratorThread-16-0:com.cloudera.cmf.model.HostInstallArgs: Deprecated option for unlimited strength J
CE. Value set to False.
2021-08-12 16:14:24,943 INFO scm-web-499421:com.cloudera.cmf.service.ServiceHandlerRegistry: Global Command GlobalHostInstall launched with
 id=1546342398
2021-08-12 16:14:24,991 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Transition
ing from INIT (PT0.049S) to CONNECT
2021-08-12 16:14:24,991 INFO NodeConfiguratorThread-16-0:net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0_14_0
2021-08-12 16:14:25,001 INFO NodeConfiguratorThread-16-0:net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
2021-08-12 16:14:25,008 INFO scm-web-499421:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/install, Status:200
2021-08-12 16:14:25,010 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:1546342398
2021-08-12 16:14:25,022 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Transitioning from CONNECT (PT0.031S) to AUTHENTICATE
2021-08-12 16:14:25,075 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Transitioning from AUTHENTICATE (PT0.053S) to MAKE_TEMP_DIR
2021-08-12 16:14:25,193 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfigurator: Executing mktemp -d /tmp/scm_prepare_node.XXXXXXXX on cldr62.ibmcc.ru
2021-08-12 16:14:25,210 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Transitioning from MAKE_TEMP_DIR (PT0.135S) to COPY_FILES
2021-08-12 16:14:25,267 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfigurator: Using default key bundle URL
2021-08-12 16:14:25,452 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.HostCertConfigurator: Creating temporary directory for certificate generation.
2021-08-12 16:14:25,525 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.HostCertConfigurator: Using host certificate generator command: /opt/cloudera/cm-agent/bin/certmanager --location /tmp/generateHostCerts16944646874431887540 gen_node_cert --output=-
2021-08-12 16:14:26,824 ERROR NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.HostCertConfigurator: Failed to generate certificates for cldr62.ibmcc.ru: 
2021-08-12 16:14:26,824 ERROR NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.HostCertConfigurator: Certificate generation failed. Temporary directory is at: /tmp/generateHostCerts16944646874431887540
2021-08-12 16:14:26,824 INFO NodeConfiguratorThread-16-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cldr62.ibmcc.ru: Setting COPY_FILES as failed and done state
2021-08-12 16:14:26,824 INFO NodeConfiguratorThread-16-0:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2021-08-12 16:14:26,824 INFO NodeConfiguratorThread-16-0:com.cloudera.cmf.model.HostInstallArgs: Deprecated option for unlimited strength JCE. Value set to False.
2021-08-12 16:14:27,774 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully.
2021-08-12 16:14:30,015 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:1546342398
2021-08-12 16:14:30,019 ERROR CommandPusher-1:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 1546342398 Failed to complete installation on host cldr62.ibmcc.ru.
2021-08-12 16:14:30,019 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546342398(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation.
2021-08-12 16:14:30,020 INFO CommandPusher-1:com.cloudera.cmf.command.components.CommandStorage: Invoked delete temp files for command:DbCommand{id=1546342398, name=GlobalHostInstall} at dir:/var/lib/cloudera-scm-server/temp/commands/1546342398

[#2 · Accepted solution]

@zinal I would request you to check if you have root privileges on the host because that's needed for Auto-TLS. Also check the permission on the directory where CM Auto-TLS stores the certificate. 

Lastly you can bypass this issue by copying certs manually from below dir:

Temporary directory is at: /tmp/generateHostCerts16944646874431887540

to the Auto-TLS dirs on the host and modify the config.ini file (refer the config.ini file from working host) and restart the agent. 

[#3]

@zinal Have you resolved your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

[#4]

Hello colleagues!

Thank you for proposing the solution, I believe that it should work - although I still had not a chance to check that.

Our HW people have ruined the test cluster, and I am still waiting for it to be recovered.

[#5]

By the way, I do not see the "Accept as solution" button - so I cannot do that.

UPDATE: had to re-login in another browser - simple "Logout" did not do the trick.

😞

[#6]

Thank you for marking the solution. It appears that you have two different community accounts. Of your two accounts, only the one that posted the question can mark the solution. If you have any further questions on the accounts or using the community, feel free to reach out to me via private message.