This host has been out of contact with the Cloudera Manager Server for too long

[#1]

Hi guys 

I face this problem every day now on at least two of the hosts 

"This host has been out of contact with the Cloudera Manager Server for too long. This host is in contact with the Host Monitor. The host's Cloudera Manager Agent version matches the Host Monitor version (5.8.1)."

• Support
• admin

Inspector Results Validations

	Inspector failed on the following hosts... ip-10-231-19-[23, 196]: Can only run host inspector when host is healthy. Inspector ran on 6 hosts.
	Individual hosts resolved their own hostnames correctly.
	No errors were found while looking for conflicting init scripts.
	No errors were found while checking /etc/hosts.
	All hosts resolved localhost to 127.0.0.1.
	All hosts checked resolved each other's hostnames correctly and in a timely manner.
	Host clocks are approximately in sync (within ten minutes).
	Host time zones are consistent across the cluster.
	No users or groups are missing.
	No conflicts detected between packages and parcels.
	No kernel versions that are known to be bad are running.
	No problems were found with /proc/sys/vm/swappiness on any of the hosts.
	No performance concerns with Transparent Huge Pages settings.
	CDH 5 Hue Python version dependency is satisfied.
	0 hosts are running CDH 4 and 8 hosts are running CDH 5.
	There are mismatched versions across the system, which will cause failures. See below for details on which hosts are running what versions of components.
	Java versions are inconsistent across the managed hosts. Check the component version below to identify hosts with inconsistent Java versions.
	All checked Cloudera Management Daemons versions are consistent with the server.
	All checked Cloudera Management Agents versions are consistent with the server.

Version Summary HostsComponent Version Release CDH Version

ml-nonprod — Group 1 (CDH 5)
ip-10-231-18-[68, 203]; ip-10-231-19-[23, 86-87, 196]
Bigtop-Tomcat (CDH 5 only)	0.7.0+cdh5.8.0+0	1.cdh5.8.0.p0.73	CDH 5
Crunch (CDH 5 only)	0.11.0+cdh5.8.0+91	1.cdh5.8.0.p0.77	CDH 5
Flume NG	1.6.0+cdh5.8.0+50	1.cdh5.8.0.p0.75	CDH 5
MapReduce 1	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
Hadoop	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HDFS	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HttpFS	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
hadoop-kms	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
MapReduce 2	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
YARN	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HBase	1.2.0+cdh5.8.0+160	1.cdh5.8.0.p0.80	CDH 5
Lily HBase Indexer	1.5+cdh5.8.0+64	1.cdh5.8.0.p0.75	CDH 5
Hive	1.1.0+cdh5.8.0+610	1.cdh5.8.0.p0.77	CDH 5
HCatalog	1.1.0+cdh5.8.0+610	1.cdh5.8.0.p0.77	CDH 5
Hue	3.9.0+cdh5.8.0+2512	1.cdh5.8.0.p0.88	CDH 5
Impala	2.6.0+cdh5.8.0+0	1.cdh5.8.0.p0.111	CDH 5
Kite (CDH 5 only)	1.0.0+cdh5.8.0+136	1.cdh5.8.0.p0.73	CDH 5
Llama (CDH 5 only)	1.0.0+cdh5.8.0+0	1.cdh5.8.0.p0.73	CDH 5
Mahout	0.9+cdh5.8.0+27	1.cdh5.8.0.p0.71	CDH 5
Oozie	4.1.0+cdh5.8.0+291	1.cdh5.8.0.p0.83	CDH 5
Parquet	1.5.0+cdh5.8.0+174	1.cdh5.8.0.p0.71	CDH 5
Pig	0.12.0+cdh5.8.0+83	1.cdh5.8.0.p0.71	CDH 5
sentry	1.5.1+cdh5.8.0+244	1.cdh5.8.0.p0.83	CDH 5
Solr	4.10.3+cdh5.8.0+423	1.cdh5.8.0.p0.79	CDH 5
spark	1.6.0+cdh5.8.0+205	1.cdh5.8.0.p0.74	CDH 5
Sqoop2	1.99.5+cdh5.8.0+38	1.cdh5.8.0.p0.72	CDH 5
Sqoop	1.4.6+cdh5.8.0+65	1.cdh5.8.0.p0.69	CDH 5
Whirr	0.9.0+cdh5.8.0+17	1.cdh5.8.0.p0.68	CDH 5
Zookeeper	3.4.5+cdh5.8.0+94	1.cdh5.8.0.p0.76	CDH 5
Cloudera Manager Management Daemons	5.8.1	1.cm581.p0.7	Not applicable
Supervisord	3.0-cm5.8.1	Unavailable	Not applicable
Java 7	JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)	Unavailable	Not applicable
Java 6	JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)	Unavailable	Not applicable
Cloudera Manager Agent	5.8.1	1.cm581.p0.7	Not applicable

HostsComponent Version Release CDH Version

ml-nonprod — Group 2 (CDH 5)
ip-10-231-19-197
Bigtop-Tomcat (CDH 5 only)	0.7.0+cdh5.8.0+0	1.cdh5.8.0.p0.73	CDH 5
Crunch (CDH 5 only)	0.11.0+cdh5.8.0+91	1.cdh5.8.0.p0.77	CDH 5
Flume NG	1.6.0+cdh5.8.0+50	1.cdh5.8.0.p0.75	CDH 5
MapReduce 1	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
Hadoop	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HDFS	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HttpFS	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
hadoop-kms	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
MapReduce 2	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
YARN	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HBase	1.2.0+cdh5.8.0+160	1.cdh5.8.0.p0.80	CDH 5
Lily HBase Indexer	1.5+cdh5.8.0+64	1.cdh5.8.0.p0.75	CDH 5
Hive	1.1.0+cdh5.8.0+610	1.cdh5.8.0.p0.77	CDH 5
HCatalog	1.1.0+cdh5.8.0+610	1.cdh5.8.0.p0.77	CDH 5
Hue	3.9.0+cdh5.8.0+2512	1.cdh5.8.0.p0.88	CDH 5
Impala	2.6.0+cdh5.8.0+0	1.cdh5.8.0.p0.111	CDH 5
Kite (CDH 5 only)	1.0.0+cdh5.8.0+136	1.cdh5.8.0.p0.73	CDH 5
Llama (CDH 5 only)	1.0.0+cdh5.8.0+0	1.cdh5.8.0.p0.73	CDH 5
Mahout	0.9+cdh5.8.0+27	1.cdh5.8.0.p0.71	CDH 5
Oozie	4.1.0+cdh5.8.0+291	1.cdh5.8.0.p0.83	CDH 5
Parquet	1.5.0+cdh5.8.0+174	1.cdh5.8.0.p0.71	CDH 5
Pig	0.12.0+cdh5.8.0+83	1.cdh5.8.0.p0.71	CDH 5
sentry	1.5.1+cdh5.8.0+244	1.cdh5.8.0.p0.83	CDH 5
Solr	4.10.3+cdh5.8.0+423	1.cdh5.8.0.p0.79	CDH 5
spark	1.6.0+cdh5.8.0+205	1.cdh5.8.0.p0.74	CDH 5
Sqoop2	1.99.5+cdh5.8.0+38	1.cdh5.8.0.p0.72	CDH 5
Sqoop	1.4.6+cdh5.8.0+65	1.cdh5.8.0.p0.69	CDH 5
Whirr	0.9.0+cdh5.8.0+17	1.cdh5.8.0.p0.68	CDH 5
Zookeeper	3.4.5+cdh5.8.0+94	1.cdh5.8.0.p0.76	CDH 5
Cloudera Manager Management Daemons	5.8.1	1.cm581.p0.7	Not applicable
Supervisord	3.0-cm5.8.1	Unavailable	Not applicable
Java 7	JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)	Unavailable	Not applicable
Java 6	JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)	Unavailable	Not applicable
Java 8	JAVA_HOME=/usr/lib/jvm/java-8-oracle java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)	Unavailable	Not applicable
Cloudera Manager Agent	5.8.1	1.cm581.p0.7	Not applicable

HostsComponent Version Release CDH Version

ml-nonprod — Group 3 (CDH 5)
ip-10-231-18-181
Bigtop-Tomcat (CDH 5 only)	0.7.0+cdh5.8.0+0	1.cdh5.8.0.p0.73	CDH 5
Crunch (CDH 5 only)	0.11.0+cdh5.8.0+91	1.cdh5.8.0.p0.77	CDH 5
Flume NG	1.6.0+cdh5.8.0+50	1.cdh5.8.0.p0.75	CDH 5
MapReduce 1	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
Hadoop	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HDFS	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HttpFS	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
hadoop-kms	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
MapReduce 2	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
YARN	2.6.0+cdh5.8.0+1601	1.cdh5.8.0.p0.93	CDH 5
HBase	1.2.0+cdh5.8.0+160	1.cdh5.8.0.p0.80	CDH 5
Lily HBase Indexer	1.5+cdh5.8.0+64	1.cdh5.8.0.p0.75	CDH 5
Hive	1.1.0+cdh5.8.0+610	1.cdh5.8.0.p0.77	CDH 5
HCatalog	1.1.0+cdh5.8.0+610	1.cdh5.8.0.p0.77	CDH 5
Hue	3.9.0+cdh5.8.0+2512	1.cdh5.8.0.p0.88	CDH 5
Impala	2.6.0+cdh5.8.0+0	1.cdh5.8.0.p0.111	CDH 5
Kite (CDH 5 only)	1.0.0+cdh5.8.0+136	1.cdh5.8.0.p0.73	CDH 5
Llama (CDH 5 only)	1.0.0+cdh5.8.0+0	1.cdh5.8.0.p0.73	CDH 5
Mahout	0.9+cdh5.8.0+27	1.cdh5.8.0.p0.71	CDH 5
Oozie	4.1.0+cdh5.8.0+291	1.cdh5.8.0.p0.83	CDH 5
Parquet	1.5.0+cdh5.8.0+174	1.cdh5.8.0.p0.71	CDH 5
Pig	0.12.0+cdh5.8.0+83	1.cdh5.8.0.p0.71	CDH 5
sentry	1.5.1+cdh5.8.0+244	1.cdh5.8.0.p0.83	CDH 5
Solr	4.10.3+cdh5.8.0+423	1.cdh5.8.0.p0.79	CDH 5
spark	1.6.0+cdh5.8.0+205	1.cdh5.8.0.p0.74	CDH 5
Sqoop2	1.99.5+cdh5.8.0+38	1.cdh5.8.0.p0.72	CDH 5
Sqoop	1.4.6+cdh5.8.0+65	1.cdh5.8.0.p0.69	CDH 5
Whirr	0.9.0+cdh5.8.0+17	1.cdh5.8.0.p0.68	CDH 5
Zookeeper	3.4.5+cdh5.8.0+94	1.cdh5.8.0.p0.76	CDH 5
Cloudera Manager Management Daemons	5.8.1	1.cm581.p0.7	Not applicable
Supervisord	3.0-cm5.8.1	Unavailable	Not applicable
Java 7	JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)	Unavailable	Not applicable
Java 6	JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)	Unavailable	Not applicable
Java 8	JAVA_HOME=/usr/lib/jvm/java-8-oracle java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)	Unavailable	Not applicable
Cloudera Manager Agent	5.8.1	1.cm581.p0.7	Not applicable

[#2]

we have faced same issue in our cluster, after investigating the cloudera agent logs we got to know their was a process got hung.hence we have restrted the cloudera agent service and the issue got resolved.

[#3]

Check if your ntp is synhronised