Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera Manager installation locks system during scm-server initialization

Highlighted

Cloudera Manager installation locks system during scm-server initialization

Explorer

I've managed to get one instance if cloudera manager up and running in Ubuntu 14.04 and 16.04, but the installer seems to lock the system whenever the scm-server service is started (e.g. when the installer opens a browser to set up at http://localhost:7180).  When I reboot the system, it freezes at the login screen for ubuntu (when I turn off quiet splash it is at the scm-manger service initialization). The install walk through i found to work is http://cloudera-cluster.blogspot.com/ 

 

The system I am running 

CPU: AMD - FX-9590 4.7GHz 8-Core OEM/Tray Processor 
Motherboard: MSI - 990FXA-GAMING ATX AM3+/AM3 Motherboard
Video Card: NVIDIA - Titan Xp 12GB Video Card 
Memory: G.Skill - Ripjaws X Series 32GB (4 x 8GB) DDR3-2133 Memory 
Storage: Western Digital - Blue 250GB 2.5" Solid State Drive   
Storage: Hitachi - Ultrastar 1TB 3.5" 7200RPM Internal Hard Drive  
 
I am able to get manager etc. installed on another similar system running an i7 with 64 gb RAM.
Thanks for any help.
 
14 REPLIES 14
Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Explorer

I managed to get past that locking somehow.  I added the repository and installed scm-server independently, then retried the installer.


Now I am stuck on the heartbeat function of the wizard.  I get the error:

 

Installation failed. Failed to receive heartbeat from agent.

  • Ensure that the host's hostname is configured properly.
  • Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
  • Ensure that ports 9000 and 9001 are not in use on the host being added.

  I have the firewall disabled and turned iptables to listen on all ports.  My hosts is set up with:

 

127.0.0.1 localhost
127.0.1.1 gpu-serv1
192.168.0.67 bigdbuffalo.com server1

 

I've tried every iteration of IP address and hostname.  

 

In the /var/log/cloudera-scm-agent/cloudera-scm-agent.log there is:

File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0$
self.exit(2)

 

 

When I run netstat -a it does appear to be listening on port 7182

tcp 0 0 *:7432 *:* LISTEN
tcp 0 0 *:7180 *:* LISTEN
tcp 0 0 *:7182 *:* LISTEN
tcp 0 0 gpu-serv1:domain *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 localhost:postgresql *:* LISTEN
tcp 0 0 localhost:5433 *:* LISTEN
tcp 0 0 bigdbuffalo.com:46764 72.21.91.97:https ESTABLISHED

Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Champion

Mate do you still have the issue ? 

well let me know the version of Java that you are runining  ?

also paste the last 50 lines of cloudera-scm-agent log using the "insert code " tool . 

Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Explorer
[02/Aug/2017 06:25:46 +0000] 16124 MainThread agent        INFO     Because agent not running as root, all processes will run with current user.
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Created /run/cloudera-scm-agent/cgroups
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/cgroups to 0751
[02/Aug/2017 06:25:48 +0000] 16124 MainThread cgroups      WARNING  Could not read capabilities from /proc/cgroups. error: sudo: no tty present and no askpass program specified

[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     No cgroups capabilities found
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Setting up supervisord event monitor.
[02/Aug/2017 06:25:48 +0000] 16124 MainThread filesystem_map INFO     Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[02/Aug/2017 06:25:48 +0000] 16124 MainThread filesystem_map INFO     Using timeout of 2.000000
[02/Aug/2017 06:25:48 +0000] 16124 MainThread filesystem_map INFO     Using join timeout of 0.100000
[02/Aug/2017 06:25:48 +0000] 16124 MainThread filesystem_map INFO     Using tolerance of 60.000000
[02/Aug/2017 06:25:48 +0000] 16124 MainThread filesystem_map INFO     Local filesystem types whitelist: ['ext2', 'ext3', 'ext4', 'xfs']
[02/Aug/2017 06:25:48 +0000] 16124 MainThread kt_renewer   INFO     Agent wide credential cache set to /run/cloudera-scm-agent/krb5cc_cm_agent_117
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Using metrics_url_timeout_seconds of 30.000000
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Using task_metrics_timeout_seconds of 5.000000
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Using max_collection_wait_seconds of 10.000000
[02/Aug/2017 06:25:48 +0000] 16124 MainThread metrics      INFO     Importing tasktracker metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/monitor/tasktracker/schema.json
[02/Aug/2017 06:25:48 +0000] 16124 MainThread ntp_monitor  INFO     Using timeout of 2.000000
[02/Aug/2017 06:25:48 +0000] 16124 MainThread dns_names    INFO     Using timeout of 30.000000
[02/Aug/2017 06:25:48 +0000] 16124 MainThread __init__     INFO     Created DNS monitor.
[02/Aug/2017 06:25:48 +0000] 16124 MainThread stacks_collection_manager INFO     Using max_uncompressed_file_size_bytes: 5242880
[02/Aug/2017 06:25:48 +0000] 16124 MainThread __init__     INFO     Importing metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/monitor/schema.json
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'SUDO': 'sudo', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Created /run/cloudera-scm-agent/process
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/process to 0751
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Created /run/cloudera-scm-agent/supervisor
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/supervisor to 0751
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Created /run/cloudera-scm-agent/flood
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/flood to 0751
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Created /run/cloudera-scm-agent/supervisor/include
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/supervisor/include to 0751
[02/Aug/2017 06:25:48 +0000] 16124 MainThread agent        ERROR    Failed to connect to previous supervisor.
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2109, in find_or_start_supervisor
    self.configure_supervisor_clients()
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2290, in configure_supervisor_clients
    supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize
    Options.realize(self, *arg, **kw)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize
    self.process_config()
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config
    self.process_config_file(do_usage)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file
    self.usage(str(msg))
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage
    self.exit(2)
SystemExit: 2
[02/Aug/2017 06:25:48 +0000] 16124 Dummy-1 daemonize    WARNING  Stopping daemon.
[02/Aug/2017 06:25:48 +0000] 16124 Dummy-1 agent        INFO     Stopping agent...

It's a fresh ubuntu spin.  When I run java –version it throws an exception.  I selected the installer to install the version it comes with.

 

This is from another system I reinstalled cloudera manager on, but the error is the same.

Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Champion

check in you system if there are Multiple Java , because I think there is another Java being installed like OpenJdk 

type in the tereminal as the below 

 

whereis java

or

/usr/lib/jvm/
make sure you dont have any OpenJDK ,if so remove it .
Restart the agents you should be fine.
if not please let me know

 

Also how are you starting the cloudera-scm-agent ? share your syntax 

Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Explorer

When I run whereisjava:

/usr/bin/java /usr/bin/X11/java /usr/share/java /usr/share/man/man1/java.1.gz

 

ls for dir /usr/lib/jvm

j2sdk1.6-oracle java-7-oracle-cloudera

 

The cloudera-scm-agent runs as a service during startup.  I went to localhost:7180 or domainame:7180 and went through the wizard, it fails on the heartbeat step.

Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Champion
Could you share your /etc/hosts and /etc/sysconfig/network
Would like to see the mapping .
Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Explorer

/etc/hosts

127.0.0.1       localhost
192.168.0.111   server1

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters


/etc/sysconfig/network is blank

 

/etc/resolvconf is:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search buffalo.rr.com

ifconfig -a | grep eth0 is:

eth0      Link encap:Ethernet  HWaddr 34:97:f6:85:9c:f2

ifcongif -a is:

eth0      Link encap:Ethernet  HWaddr 34:97:f6:85:9c:f2
          inet addr:192.168.0.111  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::3697:f6ff:fe85:9cf2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2234879 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1134086 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3307686574 (3.3 GB)  TX bytes:81231034 (81.2 MB)
          Interrupt:16 Memory:df200000-df220000

 

I have tried specifying a full domain in the /etc/hosts file e.g. 192.168.0.111    fqdn     shortname but this doesn't seem to work either.

 

Also, I tried all of the iterations within the wizard I can think of (parcels vs packages with varying versions of CDH, single user vs not, +/- installing java, root ssh vs username/pw).  Every time I get the same failed heartbeat warning.

Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Champion

Your Host file should be like this 

 

/etc/hosts

192.168.121.1    server1

 

Enter the hostname as server1

 

/etc/hostname

Restart the network

I dont have a ubuntu machine this should fix the problem , if not please let me know we can narrow down fruther

Highlighted

Re: Cloudera Manager installation locks system during scm-server initialization

Explorer
That is the format of my /etc/hosts file. The ip of the system is 192.168.0.111, should I set it to 192.168.122.1?

The /etc/hostname has server1 in there
Don't have an account?
Coming from Hortonworks? Activate your account here