Adding new host to cloudera manager cluster fails

[#1]

Am trying to add a new host to a cloudera managed cluster using wizard ... but it keeps failing with message "/tmp/scm_prepare_node.fj3OyPGR
Connection reset "

and server logs shows ...

2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GlobalHostInstall GlobalHostInstallCommandArgs{sshPort=22, userName=sw, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, parallelInstallCount=10, cmRepoUrl=null, gpgKeyCustomUrl=null, gpgKeyOverrideBundle=<none>, unlimitedJCE=true, javaInstallStrategy=NONE, agentUserMode=ROOT, cdhVersion=-1, cdhRelease=NONE>, cdhRepoUrl=null, buildCertCommand=, sslCertHostname=null, reqId=20, skipPackageInstall=false, skipCloudConfig=false, hosts=[hadoop.localdomain.38], existingHosts=[]}.
2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Install on 1 hosts.
2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Install on hadoop.localdomain.38.
2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.server.cmf.node.NodeConfiguratorService: Adding password-based configurator for hadoop.localdomain.38
2019-03-06 19:37:18,075 INFO scm-web-300:com.cloudera.server.cmf.node.NodeConfiguratorService: Submitted configurator for hadoop.localdomain.38 with id 21
2019-03-06 19:37:18,078 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Transitioning from INIT (PT0.004S) to CONNECT
2019-03-06 19:37:18,080 INFO scm-web-300:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/install, Status:200
2019-03-06 19:37:18,080 INFO NodeConfiguratorThread-20-0:net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0_14_0
2019-03-06 19:37:18,093 INFO NodeConfiguratorThread-20-0:net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
2019-03-06 19:37:18,124 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Transitioning from CONNECT (PT0.046S) to AUTHENTICATE
2019-03-06 19:37:18,203 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Transitioning from AUTHENTICATE (PT0.078S) to MAKE_TEMP_DIR
2019-03-06 19:37:18,249 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfigurator: Executing mktemp -d /tmp/scm_prepare_node.XXXXXXXX on hadoop.localdomain.38
2019-03-06 19:37:18,299 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Transitioning from MAKE_TEMP_DIR (PT0.097S) to COPY_FILES
2019-03-06 19:37:18,500 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfigurator: Using key bundle from URL: https://archive.cloudera.com/cm6/6.0.1/allkeys.asc
2019-03-06 19:37:18,653 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Setting COPY_FILES as failed and done state
2019-03-06 19:37:18,653 INFO NodeConfiguratorThread-20-0:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2019-03-06 19:37:23,105 ERROR CommandPusher:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 3767 Failed to complete installation on host hadoop.localdomain.38.
2019-03-06 19:37:23,105 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 3767(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation.
2019-03-06 19:37:23,490 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (3 skipped) AgentAvroServlet: heartbeat processing stats: average=13ms, min=10ms, max=39ms.
2019-03-06 19:37:23,655 INFO scm-web-465:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogressdata.json
2019-03-06 19:37:23,656 INFO scm-web-465:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogressdata.json, Status:200
2019-03-06 19:37:53,438 INFO scm-web-463:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogress
2019-03-06 19:37:53,439 INFO scm-web-463:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogress, Status:200

what could be the issue ?

[#2]

The issue appears to crop up when distributing certain configuration files to prepare for installing packages. Could you check or share what the failure is via the log files present under /tmp/scm_prepare_node.*/*?

[#3]

there is no log files under /tmp/scm_prepare_node
only those... local_policy.jar.8 , scm_prepare_node.sh , US_export_policy.jar.8

[#4 · Accepted solution]

Issue was master node couldn't reach https://archive.cloudera.com/, Issue was solved when IT allowed it