Support Questions
Find answers, ask questions, and share your expertise

Failed to add host

Re: Failed to add host

Super Guru

Oh, and if you wanted to just get this working with minimal effort, disable "TLS Authentication of Agents to Server" and restart Cloudera Manager with service cloudera-scm-server restart.

The existing agent configuration has a configuration that supports TLS use and certificate verification on the agent side.

Re: Failed to add host

Explorer

I think previously. "TLS authentication for agent" was not checked and that's why edge node was connected to manager properly. 

I have disabled/unchecked again and restarted cloudera-scm-agent. Now cloudera agent log on edge node changed altogether. Seems more cleaned log except one error. I things are set now edge node host should visible to cloudera manager?

Error

 Monitor-HostMonitor throttling_logger ERROR    Could not find local file system for /var/run/cloudera-scm-agent/process

 

Detailed Error

[20/Sep/2018 13:29:41 +0000] 77125 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/kerberos/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[20/Sep/2018 13:29:41 +0000] 77125 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[20/Sep/2018 13:29:41 +0000] 77125 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
[20/Sep/2018 13:29:41 +0000] 77125 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[20/Sep/2018 13:29:41 +0000] 77125 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
[20/Sep/2018 13:29:41 +0000] 77125 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[20/Sep/2018 13:29:42 +0000] 77125 MainThread agent INFO Supervisor version: 3.0, pid: 31071
[20/Sep/2018 13:29:42 +0000] 77125 MainThread agent INFO Connecting to previous supervisor: agent-31043-1537253117.
[20/Sep/2018 13:29:42 +0000] 77125 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes.
[20/Sep/2018 13:29:42 +0000] 77125 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes.
[20/Sep/2018 13:29:42 +0000] 77125 MainThread _cplogging INFO [20/Sep/2018:13:29:42] ENGINE Bus STARTING
[20/Sep/2018 13:29:42 +0000] 77125 MainThread _cplogging INFO [20/Sep/2018:13:29:42] ENGINE Started monitor thread '_TimeoutMonitor'.
[20/Sep/2018 13:29:42 +0000] 77125 MainThread _cplogging INFO [20/Sep/2018:13:29:42] ENGINE Serving on myucbpaabdapp25.cimbmy.cimbdomain.com:9000
[20/Sep/2018 13:29:42 +0000] 77125 MainThread _cplogging INFO [20/Sep/2018:13:29:42] ENGINE Bus STARTED
[20/Sep/2018 13:29:42 +0000] 77125 MainThread __init__ INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x3919050>,)
[20/Sep/2018 13:29:42 +0000] 77125 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('HostMonitor',)
[20/Sep/2018 13:29:42 +0000] 77125 MainThread agent INFO Setting default socket timeout to 45
[20/Sep/2018 13:29:42 +0000] 77125 Monitor-HostMonitor network_interfaces INFO NIC iface eth0 doesn't support ETHTOOL (95)
[20/Sep/2018 13:29:42 +0000] 77125 Monitor-HostMonitor throttling_logger ERROR Could not find local file system for /var/run/cloudera-scm-agent/process
[20/Sep/2018 13:29:42 +0000] 77125 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.10 min:0.10 mean:0.10 max:0.10 LIFE_MAX:0.10
[20/Sep/2018 13:29:42 +0000] 77125 MainThread agent INFO CM server guid: e157e5cc-09e9-4196-bac0-d396d5c1a920
[20/Sep/2018 13:29:42 +0000] 77125 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
[20/Sep/2018 13:29:42 +0000] 77125 MainThread parcel INFO Agent does create users/groups and apply file permissions
[20/Sep/2018 13:29:42 +0000] 77125 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[20/Sep/2018 13:29:42 +0000] 77125 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[20/Sep/2018 13:29:42 +0000] 77125 MainThread agent INFO Flood daemon (re)start attempt
[20/Sep/2018 13:29:43 +0000] 77125 MainThread agent INFO Triggering supervisord update.
[20/Sep/2018 13:29:44 +0000] 77125 MainThread firehoses INFO Reporting interval updated: 5.0 -> 60
[20/Sep/2018 13:29:44 +0000] 77125 MainThread agent INFO Active parcel list updated; recalculating component info.
[20/Sep/2018 13:29:44 +0000] 77125 MainThread throttling_logger INFO Identified java component java8 with full version JAVA_HOME=/usr/java/default java version "1.8.0_171" Java(TM) SE Runtime Environment (build 1.8.0_171-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode) for requested version .
[20/Sep/2018 13:30:42 +0000] 77125 MonitorDaemon-Reporter firehoses INFO Creating a connection to the ACTIVITYMONITOR.
[20/Sep/2018 13:30:42 +0000] 77125 MonitorDaemon-Reporter firehoses INFO Creating a connection to the SERVICEMONITOR.
[20/Sep/2018 13:30:42 +0000] 77125 MonitorDaemon-Reporter firehoses INFO Creating a connection to the HOSTMONITOR.

Re: Failed to add host

Super Guru

@xBigDatax,

 

That error doesn't indicate anything fatal... Chec the Hosts tab of Cloudera Manager to view all hosts.  If you see heartbeats within the last 15 seconds, all should be well with the agent communication.

 

That said, I haven't seen this in a while, but check your /etc/cloudera-scm-agent/config.ini and make sure you have this set:

 

monitored_nodev_filesystem_types=nfs,nfs4,tmpfs

 

In older config.ini files it wasn't set which resulted in seeing the Could not find local file system for /var/run/cloudera-scm-agent/process error.  If you edit config.ini, make sure to restart the agent with "service cloudera-scm-agent restart.

 

Also run "df" to make sure you see the dir mounted and that it exists.:

For example:

cm_processes    13404764   142184  13262580   2% /var/run/cloudera-scm-agent/process

 

Re: Failed to add host

Explorer

I have updated config.ini file and restarted agent. However same error. Nothing new error.

 Could not find local file system for /var/run/cloudera-scm-agent/process

 

Checked using df -h its there. I can see its mounted. Also hosttab of CM still not reflect the edge nodes.

 

Re: Failed to add host

Super Guru

@xBigDatax,

 

Let's have a look at your config.ini and "mount -l"

 

# grep -v "^#" /etc/cloudera-scm-agent/config.ini | grep -v "^$"

# mount -l

 

 

Re: Failed to add host

Explorer

Here is output. Wondering if we can connect over webex? [root@myumyhost25 cloudera-scm-agent]# grep -v "^#" /etc/cloudera-scm-agent/config.ini | grep -v "^$" [General] server_host=myumyhost03 server_port=7182 [Security] use_tls=1 verify_cert_file=/opt/cloudera/security/x509/agents.pem monitored_nodev_filesystem_types=nfs,nfs4,tmpfs [root@myumyhost25 cloudera-scm-agent]# mount -l /dev/mapper/vg_myumyhost-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/xvdb1 on /boot type ext4 (rw) /dev/mapper/vg_data-lv_data on /data type ext4 (rw) /dev/mapper/vg_data-lv_home on /home type ext4 (rw) /dev/mapper/vg_data-lv_opt on /opt type ext4 (rw) /dev/mapper/vg_data-lv_var on /var type ext4 (rw) /dev/mapper/vg_myumyhost-lv_var_crash on /var/crash type ext4 (rw) /dev/mapper/vg_myumyhost-lv_var_log on /var/log type ext4 (rw) /iso/OEL6.9/V860937-01.iso on /var/OSimage/OL6.9_x86_64 type iso9660 (ro,loop=/dev/loop0) [OL6.9 x86_64 Disc 1 20170324] none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/blkio type cgroup (rw,blkio) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpuacct type cgroup (rw,cpuacct) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpu type cgroup (rw,cpu) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/memory type cgroup (rw,memory) cm_processes on /var/run/cloudera-scm-agent/process type tmpfs (rw,mode=0751,rootcontext="unconfined_u:object_r:var_run_t:s0") [root@myumyhost25 cloudera-scm-agent]#

Re: Failed to add host

Explorer

Tell me about your environment.  I'll try to help as best I can. 

 

Some of the issues I came across while setting up the cluster was using a wildcard cert instead of single certs for each node and I had to setup a DNS server on the head node.