Unable to complete Add Host Wizard on 6.2 Manager after configuring first host.

[#1]

I have gone through the installation procedure several times trying to create 3 hosts at once but went back and decided to get the first host up.    This first host is up and I am trying to add additonal hosts.  On the wizard, Heardbeat is working, on the Install Parcels screen it shows CDH 6.2.0-1.cdh6.2.0.p0.967373 being downloaded/distributed/upacked/activated.

The issue is on the Host Inspector.

1 hosts are reporting with NONE CDH version

Also, none of the cloudera system accounts have been created on the new host.
i.e. The user hdfs is missing on the following host

The agent seems to be running and it shows up as Managed when you show all hosts in the Manager.
[root@usnj1cldrawrkr2 ~]# systemctl | grep cloudera
  cloudera-scm-agent.service                                                          loaded    active running   Cloudera Manager Agent Service

I was able to get one other host configured correctly by manually installing and then removing the cloudera services and then pushing out again with the host wizard, but no luck again. Any ideas?

[#2 · Accepted solution]

Could you share your CM agent logs snippets from right after the parcel activated and the host inspector showed the missing components/users?

The users are typically created (if they do not pre-exist) by the Cloudera Manager agent when the parcel is activated for the first time.

It is possible something may have gone wrong at that step, so having the agent logs will be helpful to troubleshoot it.

[#3]

I did a few more manual installs and uninstalls of the agent package on the target host and then ran the Wizard again from the CDH console and finally the system accounts and agent appear properly.  We may be adding more nodes in the near future and expect to see this problem again, but until then I won't be able to duplicate the problem and report back.

[#4]

Hello Harsh J,

I am having similler or same issue.

Trying to add a new host ...actually it is the first host (after the CM) to a new cluster.

But I am ending with this error 

"Command 30(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation"

The following is the "/var/log/cloudera-scm-server/cloudera-scm-server.log" since I started the process.

10:58:25.348 AM INFO ServiceHandlerRegistry Executing command GlobalHostInstall GlobalHostInstallCommandArgs{sshPort=22, userName=butmah, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, parallelInstallCount=10, cmRepoUrl=null, gpgKeyCustomUrl=null, gpgKeyOverrideBundle=<none>, unlimitedJCE=false, javaInstallStrategy=NONE, agentUserMode=ROOT, cdhVersion=-1, cdhRelease=NONE, cdhRepoUrl=null, buildCertCommand=, sslCertHostname=null, reqId=25, skipPackageInstall=false, skipCloudConfig=false, proxyProtocol=HTTP, proxyServer=10.4.32.3, proxyPort=8080, proxyUserName=null, proxyPassword=REDACTED, hosts=[node1, node2, node3, node4, node5], existingHosts=[]}.

 
10:58:25.349 AM INFO CmdStep Executing command work: Execute 1 steps in sequence

 
10:58:25.349 AM INFO CmdStep Executing command work: Install on 1 hosts.

 
10:58:25.349 AM INFO CmdStep Executing command work: Install on node5.

 
10:58:25.350 AM INFO NodeConfiguratorService Adding password-based configurator for node5

 
10:58:25.350 AM INFO NodeConfiguratorService Submitted configurator for node5 with id 30

 
10:58:25.357 AM INFO NodeConfiguratorProgress node5: Transitioning from INIT (PT0.008S) to CONNECT

 
10:58:25.359 AM INFO TransportImpl Client identity string: SSH-2.0-SSHJ_0_14_0

 
10:58:25.361 AM INFO JavaMelodyFacade Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/installretry.json, Status:200

 
10:58:25.368 AM INFO TransportImpl Server identity string: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8

 
10:58:25.422 AM INFO NodeConfiguratorProgress node5: Transitioning from CONNECT (PT0.065S) to AUTHENTICATE

 
10:58:25.474 AM INFO NodeConfiguratorProgress node5: Transitioning from AUTHENTICATE (PT0.052S) to MAKE_TEMP_DIR

 
10:58:25.616 AM INFO NodeConfigurator Executing mktemp -d /tmp/scm_prepare_node.XXXXXXXX on node5

 
10:58:25.620 AM INFO NodeConfiguratorProgress node5: Transitioning from MAKE_TEMP_DIR (PT0.146S) to COPY_FILES

 
10:58:25.719 AM INFO NodeConfigurator Using key bundle from URL: https://archive.cloudera.com/cm6/6.2.0/allkeys.asc

 
10:58:26.063 AM INFO NodeConfiguratorProgress node5: Transitioning from COPY_FILES (PT0.443S) to CHMOD

 
10:58:26.068 AM INFO NodeConfigurator Executing chmod a+x /tmp/scm_prepare_node.vQZe0yDf/scm_prepare_node.sh on node5

 
10:58:26.074 AM INFO NodeConfiguratorProgress node5: Transitioning from CHMOD (PT0.011S) to EXECUTE_SCRIPT

 
10:58:26.142 AM INFO NodeConfigurator Executing bash -c 'bash /tmp/scm_prepare_node.vQZe0yDf/scm_prepare_node.sh  --server_version 6.2.0 --server_build 968826 --packages /tmp/scm_prepare_node.vQZe0yDf/packages.scm --always /tmp/scm_prepare_node.vQZe0yDf/always_install.scm --x86_64 /tmp/scm_prepare_node.vQZe0yDf/x86_64_packages.scm --certtar /tmp/scm_prepare_node.vQZe0yDf/cert.tar --unlimitedJCE false --javaInstallStrategy NONE --agentUserMode ROOT --cm https://archive.cloudera.com/cm6/6.2.0 --skipCloudConfig false | tee /tmp/scm_prepare_node.vQZe0yDf/scm_prepare_node.log; exit ${PIPESTATUS[0]}' on node5

 
10:58:27.145 AM INFO NodeConfiguratorProgress node5: Transitioning from EXECUTE_SCRIPT (PT1.071S) to SCRIPT_START

 
10:58:27.145 AM INFO NodeConfiguratorProgress node5: Transitioning from SCRIPT_START (PT0S) to TAKE_LOCK

 
10:58:27.145 AM INFO NodeConfiguratorProgress node5: Transitioning from TAKE_LOCK (PT0S) to DETECT_ROOT

 
10:58:27.145 AM INFO NodeConfiguratorProgress node5: Transitioning from DETECT_ROOT (PT0S) to DETECT_DISTRO

 
10:58:27.145 AM INFO NodeConfiguratorProgress node5: Transitioning from DETECT_DISTRO (PT0S) to DETECT_SCM

 
10:58:28.146 AM INFO NodeConfiguratorProgress node5: Transitioning from DETECT_SCM (PT1.001S) to REPO_INSTALL

 
10:58:28.146 AM INFO NodeConfiguratorProgress node5: Transitioning from REPO_INSTALL (PT0S) to REFRESH_METADATA

 
10:58:30.354 AM INFO JavaMelodyFacade Entering HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogressdata.json

 
10:58:30.355 AM INFO JavaMelodyFacade Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogressdata.json, Status:200

 
10:58:32.079 AM INFO AgentAvroServlet (3 skipped) AgentAvroServlet: heartbeat processing stats: average=2ms, min=1ms, max=34ms.

 
10:58:32.150 AM INFO NodeConfiguratorProgress node5: Transitioning from REFRESH_METADATA (PT4.004S) to PACKAGE_INSTALL cloudera-manager-agent

 
10:58:32.160 AM WARN NodeConfigurator Command bash -c 'bash /tmp/scm_prepare_node.vQZe0yDf/scm_prepare_node.sh  --server_version 6.2.0 --server_build 968826 --packages /tmp/scm_prepare_node.vQZe0yDf/packages.scm --always /tmp/scm_prepare_node.vQZe0yDf/always_install.scm --x86_64 /tmp/scm_prepare_node.vQZe0yDf/x86_64_packages.scm --certtar /tmp/scm_prepare_node.vQZe0yDf/cert.tar --unlimitedJCE false --javaInstallStrategy NONE --agentUserMode ROOT --cm https://archive.cloudera.com/cm6/6.2.0 --skipCloudConfig false | tee /tmp/scm_prepare_node.vQZe0yDf/scm_prepare_node.log; exit ${PIPESTATUS[0]}' on node5 finished with exit status 1

 
10:58:32.160 AM INFO NodeConfiguratorProgress node5: Setting PACKAGE_INSTALL cloudera-manager-agent as failed and done state

 
10:58:32.160 AM INFO TransportImpl Disconnected - BY_APPLICATION

 
10:58:35.363 AM INFO JavaMelodyFacade Entering HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogressdata.json

 
10:58:35.364 AM INFO JavaMelodyFacade Exiting HTTP Operation: Method:POST, Path:/add-hosts-wizard/installprogressdata.json, Status:200

 
10:58:35.369 AM INFO JavaMelodyFacade Entering HTTP Operation: Method:POST, Path:/express-wizard/updateHostsState

 
10:58:35.370 AM INFO JavaMelodyFacade Exiting HTTP Operation: Method:POST, Path:/express-wizard/updateHostsState, Status:200

 
10:58:35.375 AM ERROR WorkOutputs CMD id: 30 Failed to complete installation on host node5.

 
10:58:35.375 AM ERROR DbCommand Command 30(GlobalHostInstall) has completed. finalstate:FINISHED, success:false, msg:Failed to complete installation.
 

[#5]

Would you be able to attach the contents of
/tmp/scm_prepare_node.vQZe0yDf/scm_prepare_node.log (or any/all
'/tmp/**/scm_prepare_node.log' files) from the host the install failed on
(node5 in this case)?

[#6]

Mr. Harsh,

please find the following as requested :

using SSH_CLIENT to get the SCM hostname: 10.4.34.22 37758 22
opening logging file descriptor
###CLOUDERA_SCM### SCRIPT_START
###CLOUDERA_SCM### TAKE_LOCK
BEGIN flock 4
END (0)
###CLOUDERA_SCM### DETECT_ROOT
effective UID is 1000
BEGIN which pbrun
END (1)
BEGIN sudo -S id
uid=0(root) gid=0(root) groups=0(root)
END (0)
Using 'sudo ' to acquire root privileges
###CLOUDERA_SCM### DETECT_DISTRO
BEGIN grep 'Ubuntu' /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
END (0)
BEGIN grep DISTRIB_CODENAME /etc/lsb-release
DISTRIB_CODENAME=xenial
END (0)
BEGIN echo DISTRIB_CODENAME=xenial | cut -d = -f 2
xenial
END (0)
###CLOUDERA_SCM### DETECT_SCM
BEGIN host -t PTR 10.4.34.22
Host 22.34.4.10.in-addr.arpa. not found: 3(NXDOMAIN)
END (1)
BEGIN which python
/usr/bin/python
END (0)
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' 10.4.34.22 7182
END (0)
BEGIN which wget
/usr/bin/wget
END (0)
BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo
END (4)
###CLOUDERA_SCM### REPO_INSTALL
Checking https://archive.cloudera.com/cm6/6.2.0/ubuntu1604/apt/dists/
Checking https://archive.cloudera.com/cm6/6.2.0/dists/
Using 
installing repository file /tmp/scm_prepare_node.vQZe0yDf/repos/ubuntu_xenial/cloudera-manager.list
repository file /tmp/scm_prepare_node.vQZe0yDf/repos/ubuntu_xenial/cloudera-manager.list installed
installing apt keys
BEGIN sudo apt-key add /tmp/scm_prepare_node.vQZe0yDf/customGPG
OK
END (0)
installing priority file /tmp/scm_prepare_node.vQZe0yDf/ubuntu_xenial
priority file /tmp/scm_prepare_node.vQZe0yDf/ubuntu_xenial installed
###CLOUDERA_SCM### REFRESH_METADATA
BEGIN sudo apt-get update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Reading package lists...
END (0)
BEGIN sudo apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Reading package lists...
END (0)
###CLOUDERA_SCM### PACKAGE_INSTALL cloudera-manager-agent
BEGIN sudo dpkg -l cloudera-manager-agent | grep -E '^ii[[:space:]]*cloudera-manager-agent[[:space:]]*'
dpkg-query: no packages found matching cloudera-manager-agent
END (1)
BEGIN sudo apt-cache show cloudera-manager-agent
E: No packages found
END (100)
cloudera-manager-agent must have Version=6.2.0 and Build=968826, exiting
closing logging file descriptor

 and this is what I have under "/tmp/scm_prepare_node.vQZe0yDf/" folder:

[#7]

Mr. Harsh,

would you please have a look at my reply, please?

thanks