Adding new host to cloudera manager cluster fails

Exor — Wed, 06 Mar 2019 17:56:46 GMT

Am trying to add a new host to a cloudera managed cluster using wizard ... but it keeps failing with message "/tmp/scm_prepare_node.fj3OyPGR
Connection reset "

and server logs shows ...

2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GlobalHostInstall GlobalHostInstallCommandArgs{sshPort=22, userName=sw, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, parallelInstallCount=10, cmRepoUrl=null, gpgKeyCustomUrl=null, gpgKeyOverrideBundle=<none>, unlimitedJCE=true, javaInstallStrategy=NONE, agentUserMode=ROOT, cdhVersion=-1, cdhRelease=NONE>, cdhRepoUrl=null, buildCertCommand=, sslCertHostname=null, reqId=20, skipPackageInstall=false, skipCloudConfig=false, hosts=[hadoop.localdomain.38], existingHosts=[]}.
2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Install on 1 hosts.
2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Install on hadoop.localdomain.38.
2019-03-06 19:37:18,074 INFO scm-web-300:com.cloudera.server.cmf.node.NodeConfiguratorService: Adding password-based configurator for hadoop.localdomain.38
2019-03-06 19:37:18,075 INFO scm-web-300:com.cloudera.server.cmf.node.NodeConfiguratorService: Submitted configurator for hadoop.localdomain.38 with id 21
2019-03-06 19:37:18,078 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Transitioning from INIT (PT0.004S) to CONNECT
2019-03-06 19:37:18,080 INFO scm-web-300:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/install, Status:200
2019-03-06 19:37:18,080 INFO NodeConfiguratorThread-20-0:net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0_14_0
2019-03-06 19:37:18,093 INFO NodeConfiguratorThread-20-0:net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
2019-03-06 19:37:18,124 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Transitioning from CONNECT (PT0.046S) to AUTHENTICATE
2019-03-06 19:37:18,203 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Transitioning from AUTHENTICATE (PT0.078S) to MAKE_TEMP_DIR
2019-03-06 19:37:18,249 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfigurator: Executing mktemp -d /tmp/scm_prepare_node.XXXXXXXX on hadoop.localdomain.38
2019-03-06 19:37:18,299 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Transitioning from MAKE_TEMP_DIR (PT0.097S) to COPY_FILES
2019-03-06 19:37:18,500 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfigurator: Using key bundle from URL: https://archive.cloudera.com/cm6/6.0.1/allkeys.asc
2019-03-06 19:37:18,653 INFO NodeConfiguratorThread-20-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: hadoop.localdomain.38: Setting COPY_FILES as failed and done state
2019-03-06 19:37:18,653 INFO NodeConfiguratorThread-20-0:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2019-03-06 19:37:23,105 ERROR CommandPusher:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 3767 Failed to complete installation on host hadoop.localdomain.38.
2019-03-06 19:37:23,105 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 3767(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation.
2019-03-06 19:37:23,490 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (3 skipped) AgentAvroServlet: heartbeat processing stats: average=13ms, min=10ms, max=39ms.
2019-03-06 19:37:23,655 INFO scm-web-465:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogressdata.json
2019-03-06 19:37:23,656 INFO scm-web-465:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogressdata.json, Status:200
2019-03-06 19:37:53,438 INFO scm-web-463:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogress
2019-03-06 19:37:53,439 INFO scm-web-463:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogress, Status:200

what could be the issue ?

Re: Adding new host to cloudera manager cluster fails

Harsh J — Thu, 07 Mar 2019 01:20:19 GMT

The issue appears to crop up when distributing certain configuration files to prepare for installing packages. Could you check or share what the failure is via the log files present under /tmp/scm_prepare_node.*/*?

Re: Adding new host to cloudera manager cluster fails

Exor — Thu, 07 Mar 2019 08:40:37 GMT

there is no log files under /tmp/scm_prepare_node
only those... local_policy.jar.8 , scm_prepare_node.sh , US_export_policy.jar.8

Re: Adding new host to cloudera manager cluster fails

Exor — Thu, 07 Mar 2019 15:46:05 GMT

Issue was master node couldn't reach https://archive.cloudera.com/, Issue was solved when IT allowed it

question Adding new host to cloudera manager cluster fails in Support Questions

Adding new host to cloudera manager cluster fails

Re: Adding new host to cloudera manager cluster fails

Re: Adding new host to cloudera manager cluster fails

Re: Adding new host to cloudera manager cluster fails