Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cluster Installation distributing parcel problem

avatar
New Contributor

Hello

 

I've been trying to install Cloudera 5.10 on Centos 7.x in a VMWare environment.  Each time Cloudera Manager gets stuck at Distributed.  Only 2 nodes are correct.

 

distribution.jpg

 

 Disk space is not a constraint

disk_space.jpg

 selinux, ipv6, firewalls are disabled

 

 

LOG: (/var/log/cloudera-scm-agent:

 

[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     SCM Agent Version: 5.10.0
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Agent Protocol Version: 4
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Using Host ID: c0dc62df-1791-496a-a70f-40eea5c1220d
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Using directory: /run/cloudera-scm-agent
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Using supervisor binary path: /usr/lib64/cmf/agent/build/env/bin/supervisord
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Agent Logging Level: INFO
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     No command line vars
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Found database jar: /usr/share/java/mysql-connector-java.jar
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
[01/Mar/2017 15:00:07 +0000] 7362 MainThread agent        INFO     Agent starting as pid 7362 user root(0) group root(0).
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Created /run/cloudera-scm-agent/cgroups
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/cgroups to 0751
[01/Mar/2017 15:00:09 +0000] 7362 MainThread cgroups      INFO     Found existing subsystem cpuacct at /sys/fs/cgroup/cpu,cpuacct
[01/Mar/2017 15:00:09 +0000] 7362 MainThread cgroups      INFO     Found existing subsystem cpu at /sys/fs/cgroup/cpu,cpuacct
[01/Mar/2017 15:00:09 +0000] 7362 MainThread cgroups      INFO     Found existing subsystem memory at /sys/fs/cgroup/memory
[01/Mar/2017 15:00:09 +0000] 7362 MainThread cgroups      INFO     Found existing subsystem blkio at /sys/fs/cgroup/blkio
[01/Mar/2017 15:00:09 +0000] 7362 MainThread cgroups      INFO     Found cgroups subsystem: cpu
[01/Mar/2017 15:00:09 +0000] 7362 MainThread cgroups      INFO     Found cgroups subsystem: cpuacct
[01/Mar/2017 15:00:09 +0000] 7362 MainThread cgroups      INFO     Found cgroups subsystem: memory
[01/Mar/2017 15:00:09 +0000] 7362 MainThread cgroups      INFO     Found cgroups subsystem: blkio
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': 8796093022207, 'default_memory_soft_limit_in_bytes': 8796093022207, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True}
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Setting up supervisord event monitor.
[01/Mar/2017 15:00:09 +0000] 7362 MainThread filesystem_map INFO     Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[01/Mar/2017 15:00:09 +0000] 7362 MainThread filesystem_map INFO     Using timeout of 2.000000
[01/Mar/2017 15:00:09 +0000] 7362 MainThread filesystem_map INFO     Using join timeout of 0.100000
[01/Mar/2017 15:00:09 +0000] 7362 MainThread filesystem_map INFO     Using tolerance of 60.000000
[01/Mar/2017 15:00:09 +0000] 7362 MainThread filesystem_map INFO     Local filesystem types whitelist: ['ext2', 'ext3', 'ext4', 'xfs']
[01/Mar/2017 15:00:09 +0000] 7362 MainThread kt_renewer   INFO     Agent wide credential cache set to /run/cloudera-scm-agent/krb5cc_cm_agent_0
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Using metrics_url_timeout_seconds of 30.000000
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Using task_metrics_timeout_seconds of 5.000000
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Using max_collection_wait_seconds of 10.000000
[01/Mar/2017 15:00:09 +0000] 7362 MainThread metrics      INFO     Importing tasktracker metric schema from file /usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/tasktracker/schema.json
[01/Mar/2017 15:00:09 +0000] 7362 MainThread ntp_monitor  INFO     Using timeout of 2.000000
[01/Mar/2017 15:00:09 +0000] 7362 MainThread dns_names    INFO     Using timeout of 30.000000
[01/Mar/2017 15:00:09 +0000] 7362 MainThread __init__     INFO     Created DNS monitor.
[01/Mar/2017 15:00:09 +0000] 7362 MainThread stacks_collection_manager INFO     Using max_uncompressed_file_size_bytes: 5242880
[01/Mar/2017 15:00:09 +0000] 7362 MainThread __init__     INFO     Importing metric schema from file /usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/monitor/schema.json
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Created /run/cloudera-scm-agent/process
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/process to 0751
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Created /run/cloudera-scm-agent/supervisor
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/supervisor to 0751
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Created /run/cloudera-scm-agent/flood
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Chowning /run/cloudera-scm-agent/flood to cloudera-scm (996) cloudera-scm (994)
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/flood to 0751
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Created /run/cloudera-scm-agent/supervisor/include
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/supervisor/include to 0751
[01/Mar/2017 15:00:09 +0000] 7362 MainThread agent        ERROR    Failed to connect to previous supervisor.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/agent.py", line 2056, in find_or_start_supervisor
    self.configure_supervisor_clients()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/agent.py", line 2235, in configure_supervisor_clients
    supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize
    Options.realize(self, *arg, **kw)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize
    self.process_config()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config
    self.process_config_file(do_usage)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file
    self.usage(str(msg))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage
    self.exit(2)
SystemExit: 2
[01/Mar/2017 15:00:09 +0000] 7362 MainThread tmpfs        INFO     Successfully mounted tmpfs at /run/cloudera-scm-agent/process
[01/Mar/2017 15:00:11 +0000] 7362 MainThread agent        INFO     Trying to connect to newly launched supervisor (Attempt 1)
[01/Mar/2017 15:00:11 +0000] 7362 MainThread agent        INFO     Supervisor version: 3.0, pid: 7391
[01/Mar/2017 15:00:11 +0000] 7362 MainThread agent        INFO     Successfully connected to supervisor
[01/Mar/2017 15:00:11 +0000] 7362 MainThread status_server INFO     Using maximum impala profile bundle size of 1073741824 bytes.
[01/Mar/2017 15:00:11 +0000] 7362 MainThread status_server INFO     Using maximum stacks log bundle size of 1073741824 bytes.
[01/Mar/2017 15:00:11 +0000] 7362 MainThread _cplogging   INFO     [01/Mar/2017:15:00:11] ENGINE Bus STARTING
[01/Mar/2017 15:00:11 +0000] 7362 MainThread _cplogging   INFO     [01/Mar/2017:15:00:11] ENGINE Started monitor thread '_TimeoutMonitor'.
[01/Mar/2017 15:00:11 +0000] 7362 MainThread _cplogging   INFO     [01/Mar/2017:15:00:11] ENGINE Serving on vm01.aposentinel.com:9000
[01/Mar/2017 15:00:11 +0000] 7362 MainThread _cplogging   INFO     [01/Mar/2017:15:00:11] ENGINE Bus STARTED
[01/Mar/2017 15:00:11 +0000] 7362 MainThread __init__     INFO     New monitor: (<cmf.monitor.host.HostMonitor object at 0x1f5af10>,)
[01/Mar/2017 15:00:11 +0000] 7362 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('HostMonitor',)
[01/Mar/2017 15:00:11 +0000] 7362 MainThread agent        INFO     Setting default socket timeout to 30
[01/Mar/2017 15:00:11 +0000] 7362 Monitor-HostMonitor network_interfaces WARNING  Interface 'eth0' reported unknown speed '4294967295'
[01/Mar/2017 15:00:11 +0000] 7362 Monitor-HostMonitor network_interfaces WARNING  Interface 'eth0' reported unknown duplex mode '255'
[01/Mar/2017 15:00:12 +0000] 7362 MainThread heartbeat_tracker INFO     HB stats (seconds): num:1 LIFE_MIN:0.24 min:0.24 mean:0.24 max:0.24 LIFE_MAX:0.24
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        INFO     CM server guid: 480c885c-27a1-4402-ae51-a415aa0d115e
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        INFO     Using parcels directory from server provided value: /opt/cloudera/parcels
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        INFO     Created /opt/cloudera/parcels
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        INFO     Chowning /opt/cloudera/parcels to root (0) root (0)
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        INFO     Chmod'ing /opt/cloudera/parcels to 0755
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        INFO     Created /opt/cloudera/parcel-cache
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        INFO     Chowning /opt/cloudera/parcel-cache to root (0) root (0)
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        INFO     Chmod'ing /opt/cloudera/parcel-cache to 0755
[01/Mar/2017 15:00:12 +0000] 7362 MainThread parcel       INFO     Agent does create users/groups and apply file permissions
[01/Mar/2017 15:00:12 +0000] 7362 MainThread downloader   INFO     Downloader path: /opt/cloudera/parcel-cache
[01/Mar/2017 15:00:12 +0000] 7362 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[01/Mar/2017 15:00:12 +0000] 7362 MainThread agent        ERROR    Caught unexpected exception in main loop.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/agent.py", line 710, in __issue_heartbeat
    self._init_after_first_heartbeat_response(resp_data)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/agent.py", line 947, in _init_after_first_heartbeat_response
    self.client_configs.load()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/client_configs.py", line 682, in load
    new_deployed.update(self._lookup_alternatives(fname))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/client_configs.py", line 432, in _lookup_alternatives
    return self._parse_alternatives(alt_name, out)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/client_configs.py", line 444, in _parse_alternatives
    path, _, _, priority_str = line.rstrip().split(" ")
ValueError: too many values to unpack

 

LOG /var/log/cloudera/scm-server/

 

2017-03-01 15:04:24,393 INFO 1215304684@agentServer-3:com.cloudera.server.cmf.AgentProtocolImpl: [DbHost{id=3, hostId=b44317fd-5f50-4019-9266-112580e860e8, hostName=vm02.aposentinel.com}] Added to rack group: /default
2017-03-01 15:04:24,397 INFO 1215304684@agentServer-3:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: com.cloudera.cmf.model.DbHost.name (#3) has changed]}.
2017-03-01 15:04:24,432 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.003S
2017-03-01 15:04:24,432 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check execution stats: average=0ms, min=0ms, max=0ms.
2017-03-01 15:04:24,507 INFO NodeConfiguratorThread-0-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm02.aposentinel.com: New state is a done state
2017-03-01 15:04:24,507 INFO NodeConfiguratorThread-0-1:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm02.aposentinel.com: Transitioning from WAIT_FOR_HEARTBEAT (PT4.025S) to COMPLETE
2017-03-01 15:04:24,507 INFO NodeConfiguratorThread-0-1:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2017-03-01 15:04:31,749 INFO NodeConfiguratorThread-0-3:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm04.aposentinel.com: Transitioning from PACKAGE_INSTALL cloudera-manager-agent (PT291.417S) to PACKAGE_INSTALL cloudera-manager-daemons
2017-03-01 15:04:32,751 INFO NodeConfiguratorThread-0-3:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm04.aposentinel.com: Transitioning from PACKAGE_INSTALL cloudera-manager-daemons (PT1.002S) to INSTALL_JCE
2017-03-01 15:04:32,751 INFO NodeConfiguratorThread-0-3:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm04.aposentinel.com: Transitioning from INSTALL_JCE (PT0S) to AGENT_CONFIGURE
2017-03-01 15:04:32,751 INFO NodeConfiguratorThread-0-3:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm04.aposentinel.com: Transitioning from AGENT_CONFIGURE (PT0S) to AGENT_START
2017-03-01 15:04:33,422 INFO NodeConfiguratorThread-0-3:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm04.aposentinel.com: Transitioning from AGENT_START (PT0.671S) to SCRIPT_SUCCESS
2017-03-01 15:04:33,422 INFO NodeConfiguratorThread-0-3:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm04.aposentinel.com: Transitioning from SCRIPT_SUCCESS (PT0S) to WAIT_FOR_HEARTBEAT
2017-03-01 15:04:37,392 INFO 1215304684@agentServer-3:com.cloudera.server.cmf.AgentProtocolImpl: Setting default rackId for host 8df2aa3d-177f-469d-9154-d5a919dd51e7: /default
2017-03-01 15:04:37,394 INFO 1215304684@agentServer-3:com.cloudera.server.cmf.AgentProtocolImpl: [DbHost{id=4, hostId=8df2aa3d-177f-469d-9154-d5a919dd51e7, hostName=vm04.aposentinel.com}] Added to rack group: /default
2017-03-01 15:04:37,400 INFO 1215304684@agentServer-3:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: com.cloudera.cmf.model.DbHost.name (#4) has changed]}.
2017-03-01 15:04:37,414 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.003S
2017-03-01 15:04:37,414 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check execution stats: average=0ms, min=0ms, max=0ms.
2017-03-01 15:04:37,457 INFO NodeConfiguratorThread-0-3:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm04.aposentinel.com: New state is a done state
2017-03-01 15:04:37,457 INFO NodeConfiguratorThread-0-3:com.cloudera.server.cmf.node.NodeConfiguratorProgress: vm04.aposentinel.com: Transitioning from WAIT_FOR_HEARTBEAT (PT4.035S) to COMPLETE
2017-03-01 15:04:37,457 INFO NodeConfiguratorThread-0-3:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2017-03-01 15:04:56,539 INFO 1748755379@scm-web-4:com.cloudera.server.cmf.components.OperationsManagerImpl: Creating cluster cluster with version CDH 5 and rounded off version CDH 5.0.0
2017-03-01 15:04:56,574 INFO 1748755379@scm-web-4:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: com.cloudera.cmf.model.DbHost.cluster (#1) has changed]}.
2017-03-01 15:04:56,592 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.004S
2017-03-01 15:04:56,592 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check execution stats: average=0ms, min=0ms, max=0ms.
2017-03-01 15:05:00,614 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelManagerImpl: Downloading parcels of CDH:5.10.0-1.cdh5.10.0.p0.41 for distros [RHEL7]
2017-03-01 15:05:00,699 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Preparing to download: https://archive.cloudera.com/cdh5/parcels/5.10/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
2017-03-01 15:06:41,370 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2017-03-01 15:06:41,391 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2015-03-02T14:06:41.371Z to reap.
2017-03-01 15:06:41,393 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2017-03-01 15:08:39,803 INFO New I/O  worker #19:com.cloudera.parcel.components.ParcelDownloaderImpl: Completed download of: https://archive.cloudera.com/cdh5/parcels/5.10/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
2017-03-01 15:08:41,790 INFO 696999056@agentServer-5:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=14ms, min=0ms, max=207ms.
2017-03-01 15:08:41,791 INFO 696999056@agentServer-5:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=1ms, min=0ms, max=33ms.
2017-03-01 15:08:48,446 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Discovered parcel on CM server: CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
2017-03-01 15:08:48,446 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Created torrent file: /opt/cloudera/parcel-repo/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.torrent
2017-03-01 15:08:48,451 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Creating single-file torrent for CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel...
2017-03-01 15:08:48,454 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashing data from CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel with 8 threads (3018 pieces)...
2017-03-01 15:08:49,051 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 10% complete
2017-03-01 15:08:49,672 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 20% complete
2017-03-01 15:08:50,253 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 30% complete
2017-03-01 15:08:50,846 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 40% complete
2017-03-01 15:08:51,463 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 50% complete
2017-03-01 15:08:52,054 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 60% complete
2017-03-01 15:08:52,635 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 70% complete
2017-03-01 15:08:53,226 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 80% complete
2017-03-01 15:08:53,822 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 90% complete
2017-03-01 15:08:54,459 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (1582136490 bytes) in 3018 pieces (3018 expected) in 6002,9ms.
2017-03-01 15:08:54,464 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Single-file torrent information:
2017-03-01 15:08:54,464 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Torrent name: CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
2017-03-01 15:08:54,464 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Announced at: Seems to be trackerless
2017-03-01 15:08:54,465 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created on..: Wed Mar 01 15:08:48 CET 2017
2017-03-01 15:08:54,465 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created by..: cm-server
2017-03-01 15:08:54,465 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Pieces......: 3018 piece(s) (524288 byte(s)/piece)
2017-03-01 15:08:54,465 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Total size..: 1 582 136 490 byte(s)
2017-03-01 15:08:54,493 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelManagerImpl: Distributing parcel CDH:5.10.0-1.cdh5.10.0.p0.41 on cluster cluster
2017-03-01 15:08:54,591 WARN 271140938@scm-web-12:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.torrent
2017-03-01 15:08:54,594 INFO 271140938@scm-web-12:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.torrent (60527 bytes) to 192.168.114.22 over PT0.002S.  Rate: 28,862 MB/s
2017-03-01 15:08:54,631 WARN 271140938@scm-web-12:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.torrent
2017-03-01 15:08:54,632 INFO 271140938@scm-web-12:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.torrent (60527 bytes) to 192.168.114.21 over PT0.001S.  Rate: 57,723 MB/s
2017-03-01 15:10:31,281 INFO 1027593378@scm-web-10:com.cloudera.server.web.cmf.ParcelController: Parcel download request: CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel from : 192.168.114.22
2017-03-01 15:10:31,292 INFO 1027593378@scm-web-10:com.cloudera.server.web.cmf.ParcelController: Served parcel piece CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel (1048576 bytes) to 192.168.114.22 over PT0.006S.  Rate: 166,667 MB/s
2017-03-01 15:12:10,730 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel CDH:5.10.0-1.cdh5.10.0.p0.41 on cluster cluster
2017-03-01 15:12:10,808 INFO ParcelUpdateService:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: com.cloudera.cmf.model.DbCluster.cdhVersionForDb (#1) has changed]}.
2017-03-01 15:12:10,864 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.011S
2017-03-01 15:12:10,864 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check execution stats: average=0ms, min=0ms, max=0ms.
2017-03-01 15:12:10,922 WARN 1918889447@scm-web-14:com.cloudera.parcel.ClusterParcelStatus: Parcel not distributed but have active state ACTIVATING
2017-03-01 15:12:10,922 WARN 1918889447@scm-web-14:com.cloudera.parcel.ClusterParcelStatus: Parcel not distributed but have active state ACTIVATING
2017-03-01 15:12:11,093 WARN 1918889447@scm-web-14:com.cloudera.parcel.ClusterParcelStatus: Parcel not distributed but have active state ACTIVATING
2017-03-01 15:12:11,093 WARN 1918889447@scm-web-14:com.cloudera.parcel.ClusterParcelStatus: Parcel not distributed but have active state ACTIVATING
2017-03-01 15:12:12,138 WARN 1918889447@scm-web-14:com.cloudera.parcel.ClusterParcelStatus: Parcel not distributed but have active state ACTIVATING
2017-03-01 15:12:12,138 WARN 1918889447@scm-web-14:com.cloudera.parcel.ClusterParcelStatus: Parcel not distributed but have active state ACTIVATING
2017-03-01 15:12:12,484 WARN 1918889447@scm-web-14:com.cloudera.parcel.ClusterParcelStatus: Parcel not distributed but have active state ACTIVATING

 

 

The same happened when I was trying to install cloudera by package, only two nodes were correct...
Any help will be appreciated

 

1 ACCEPTED SOLUTION

avatar
Champion

Check out this thread.  There seems to be an issue with python getting the wrong number of columns from the alternatives command.  The likely culprit is OpenJDK and if you can uninstall it but it could be something else.  The thread contains a command to hunt down the exact issue.

 

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Problem-with-cloudera-agent/td-p/476...

View solution in original post

2 REPLIES 2

avatar
Champion

Check out this thread.  There seems to be an issue with python getting the wrong number of columns from the alternatives command.  The likely culprit is OpenJDK and if you can uninstall it but it could be something else.  The thread contains a command to hunt down the exact issue.

 

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Problem-with-cloudera-agent/td-p/476...

avatar
New Contributor

Thanks,

 

Removing all alternatives OpenJDK helped:)