Inspect All hosts, Inspector failed on the following hosts...

[#1]

i have installed CDH-6.3.0 , with 5 nodes (Cent os 7) but after installing cloudera manager when i tried to inspect all hosts it only ran on one host as you see below  

here is inspect validation results

###############################################################################
 ##############################################################################

 ##############################################################################

And here is /etc/hosts file for server and one of the agents 

Also my firewalld service is disabled.

i don't know what is wrong with my installation, if anyone can help me please !! 

Thanks in advance.

[#2]

@dv_conan Make sure the agent is heart beating in the CM > Hosts > All hosts page.

Then check the agent logs (/var/log/cloudera-scm-agent/cloudera-scm-agent.log) and see if there is any error.

[#3]

@GangWar i checked all possible logs but i couldn't find the error yet 
here is (/var/log/cloudera-scm-agent/cloudera-scm-agent.log)

[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Created /var/run/cloudera-scm-agent/process/606-cluster-host-inspector
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Chowning /var/run/cloudera-scm-agent/process/606-cluster-host-inspector to root (0) root (0)
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Chmod'ing /var/run/cloudera-scm-agent/process/606-cluster-host-inspector to 0751
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Created /var/run/cloudera-scm-agent/process/606-cluster-host-inspector/logs
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Chowning /var/run/cloudera-scm-agent/process/606-cluster-host-inspector/logs to root (0) root (0)
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Chmod'ing /var/run/cloudera-scm-agent/process/606-cluster-host-inspector/logs to 0751
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 process	  INFO     [606-cluster-host-inspector] Refreshing process files: None
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 parcel	  INFO     prepare_environment begin: {}, [], []
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 parcel	  INFO     No parcels activated for use
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 __init__     INFO     Extracted 1 files and 0 dirs to /var/run/cloudera-scm-agent/process/606-cluster-host-inspector.
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 process	  INFO     [606-cluster-host-inspector] Launching process. one-off True, command mgmt/mgmt.sh, args [u'inspector', u'input.json', u'output.json', u$
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 supervisor   WARNING  Failed while getting process info. Retrying. (<Fault 10: 'BAD_NAME: 606-cluster-host-inspector'>)
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 supervisor   INFO     Triggering supervisord update.
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 util         INFO     Using generic audit plugin for process cluster-host-inspector
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 util         INFO     Creating metadata plugin for process cluster-host-inspector
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 util         INFO     Using specific metadata plugin for process cluster-host-inspector
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 util         INFO     Using generic metadata plugin for process cluster-host-inspector
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 process	  INFO     Begin audit plugin refresh
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 throttling_logger INFO     (3 skipped) Scheduling a refresh for Audit Plugin for cluster-host-inspector with count 1 pipelines names [''].
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 process	  INFO     Begin metadata plugin refresh
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 process	  INFO     Not creating a monitor for 606-cluster-host-inspector: should_monitor returns false
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 process	  INFO     Daemon refresh complete for process 606-cluster-host-inspector.
[26/Jun/2020 16:38:13 +0000] 25076 Audit-Plugin navigator_plugin INFO     Pipelines updated for Audit Plugin: []
[26/Jun/2020 16:38:13 +0000] 25076 Audit-Plugin throttling_logger INFO     (3 skipped) Refreshing Audit Plugin for cluster-host-inspector with count 0 pipelines names [].
[26/Jun/2020 16:38:14 +0000] 25076 Metadata-Plugin navigator_plugin INFO     Pipelines updated for Metadata Plugin: []
[26/Jun/2020 16:38:14 +0000] 25076 MainThread agent        INFO     PID '27699' associated with process '606-cluster-host-inspector' with payload 'processname:606-cluster-host-inspector groupname:606-clu$
[26/Jun/2020 16:38:14 +0000] 25076 MainThread agent        WARNING  Process exited event for process '606-cluster-host-inspector' does not have a monitor to handle event - the event will be ignored.
[26/Jun/2020 16:38:14 +0000] 25076 MainThread process	   INFO     [606-cluster-host-inspector] Unregistered supervisor process EXITED
[26/Jun/2020 16:38:16 +0000] 25076 MainThread supervisor   INFO     Triggering supervisord update.
[26/Jun/2020 16:38:29 +0000] 25076 Thread-17 process	  INFO     [606-cluster-host-inspector] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[26/Jun/2020 16:38:29 +0000] 25076 Thread-17 process	  INFO     [606-cluster-host-inspector] Deactivating process (skipped)
[26/Jun/2020 16:38:29 +0000] 25076 Thread-17 process	  INFO     [606-cluster-host-inspector] stopping monitors
[26/Jun/2020 16:38:33 +0000] 25076 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for cluster-host-inspector with count 0 pipelines names [].
[26/Jun/2020 16:38:34 +0000] 25076 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for cluster-host-inspector with count 0 pipelines names [].
[26/Jun/2020 16:38:59 +0000] 25076 MainThread heartbeat_tracker INFO     HB stats (seconds): num:42 LIFE_MIN:0.01 min:0.01 mean:0.02 max:0.05 LIFE_MAX:0.09

and here is stderr.log

+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/626-cluster-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-connector.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ echo_and_exec /usr/java/jdk1.8.0_181-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/626-cluster-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-connector.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
+ echo 'Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java' -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/626-cluster-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-connector.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
+ exec /usr/java/jdk1.8.0_181-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/626-cluster-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-connector.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
Error: Could not find or load main class com.cloudera.cmf.inspector.Inspector

[#4]

@dv_conan  The error is due to missing class form the agent.

Error: Could not find or load main class com.cloudera.cmf.inspector.Inspector

  You have to install again and do a restart. 

yum install cloudera-manager-daemons
service cloudera-scm-agent restart

This should resolve the issue. 

[#5]

- Make sure in the phase of Install Agent , the heart beat test passed.

- Make sure each host has cm_guid.

[root@redhat1 cloudera-scm-agent]# pwd
/var/lib/cloudera-scm-agent
[root@redhat1 cloudera-scm-agent]# ls
active_parcels.json cm_guid response.avro uuid
[root@redhat1 cloudera-scm-agent]#