Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Inspect All hosts, Inspector failed on the following hosts...

avatar
Explorer

i have installed CDH-6.3.0 , with 5 nodes (Cent os 7) but after installing cloudera manager when i tried to inspect all hosts it only ran on one host as you see below  

 

here is inspect validation results

cloudera-questions-1.png

###############################################################################cloudera-questions-2.png
 ##############################################################################

cloudera-questions-3.png

 ##############################################################################

 

And here is /etc/hosts file for server and one of the agents 

cloudera-questions-4.png

 

cloudera-questions-5.png

 

Also my firewalld service is disabled.

i don't know what is wrong with my installation, if anyone can help me please !! 

 

Thanks in advance.

 

4 REPLIES 4

avatar
Master Guru

@dv_conan Make sure the agent is heart beating in the CM > Hosts > All hosts page.

Then check the agent logs (/var/log/cloudera-scm-agent/cloudera-scm-agent.log) and see if there is any error.


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Explorer

@GangWar i checked all possible logs but i couldn't find the error yet 
here is (/var/log/cloudera-scm-agent/cloudera-scm-agent.log)

 

 

[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Created /var/run/cloudera-scm-agent/process/606-cluster-host-inspector
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Chowning /var/run/cloudera-scm-agent/process/606-cluster-host-inspector to root (0) root (0)
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Chmod'ing /var/run/cloudera-scm-agent/process/606-cluster-host-inspector to 0751
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Created /var/run/cloudera-scm-agent/process/606-cluster-host-inspector/logs
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Chowning /var/run/cloudera-scm-agent/process/606-cluster-host-inspector/logs to root (0) root (0)
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 agent        INFO     Chmod'ing /var/run/cloudera-scm-agent/process/606-cluster-host-inspector/logs to 0751
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 process	  INFO     [606-cluster-host-inspector] Refreshing process files: None
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 parcel	  INFO     prepare_environment begin: {}, [], []
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 parcel	  INFO     No parcels activated for use
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 __init__     INFO     Extracted 1 files and 0 dirs to /var/run/cloudera-scm-agent/process/606-cluster-host-inspector.
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 process	  INFO     [606-cluster-host-inspector] Launching process. one-off True, command mgmt/mgmt.sh, args [u'inspector', u'input.json', u'output.json', u$
[26/Jun/2020 16:38:09 +0000] 25076 Thread-14 supervisor   WARNING  Failed while getting process info. Retrying. (<Fault 10: 'BAD_NAME: 606-cluster-host-inspector'>)
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 supervisor   INFO     Triggering supervisord update.
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 util         INFO     Using generic audit plugin for process cluster-host-inspector
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 util         INFO     Creating metadata plugin for process cluster-host-inspector
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 util         INFO     Using specific metadata plugin for process cluster-host-inspector
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 util         INFO     Using generic metadata plugin for process cluster-host-inspector
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 process	  INFO     Begin audit plugin refresh
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 throttling_logger INFO     (3 skipped) Scheduling a refresh for Audit Plugin for cluster-host-inspector with count 1 pipelines names [''].
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 process	  INFO     Begin metadata plugin refresh
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 process	  INFO     Not creating a monitor for 606-cluster-host-inspector: should_monitor returns false
[26/Jun/2020 16:38:11 +0000] 25076 Thread-14 process	  INFO     Daemon refresh complete for process 606-cluster-host-inspector.
[26/Jun/2020 16:38:13 +0000] 25076 Audit-Plugin navigator_plugin INFO     Pipelines updated for Audit Plugin: []
[26/Jun/2020 16:38:13 +0000] 25076 Audit-Plugin throttling_logger INFO     (3 skipped) Refreshing Audit Plugin for cluster-host-inspector with count 0 pipelines names [].
[26/Jun/2020 16:38:14 +0000] 25076 Metadata-Plugin navigator_plugin INFO     Pipelines updated for Metadata Plugin: []
[26/Jun/2020 16:38:14 +0000] 25076 MainThread agent        INFO     PID '27699' associated with process '606-cluster-host-inspector' with payload 'processname:606-cluster-host-inspector groupname:606-clu$
[26/Jun/2020 16:38:14 +0000] 25076 MainThread agent        WARNING  Process exited event for process '606-cluster-host-inspector' does not have a monitor to handle event - the event will be ignored.
[26/Jun/2020 16:38:14 +0000] 25076 MainThread process	   INFO     [606-cluster-host-inspector] Unregistered supervisor process EXITED
[26/Jun/2020 16:38:16 +0000] 25076 MainThread supervisor   INFO     Triggering supervisord update.
[26/Jun/2020 16:38:29 +0000] 25076 Thread-17 process	  INFO     [606-cluster-host-inspector] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[26/Jun/2020 16:38:29 +0000] 25076 Thread-17 process	  INFO     [606-cluster-host-inspector] Deactivating process (skipped)
[26/Jun/2020 16:38:29 +0000] 25076 Thread-17 process	  INFO     [606-cluster-host-inspector] stopping monitors
[26/Jun/2020 16:38:33 +0000] 25076 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for cluster-host-inspector with count 0 pipelines names [].
[26/Jun/2020 16:38:34 +0000] 25076 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for cluster-host-inspector with count 0 pipelines names [].
[26/Jun/2020 16:38:59 +0000] 25076 MainThread heartbeat_tracker INFO     HB stats (seconds): num:42 LIFE_MIN:0.01 min:0.01 mean:0.02 max:0.05 LIFE_MAX:0.09

 

and here is stderr.log

+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/626-cluster-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-connector.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:'
+ echo_and_exec /usr/java/jdk1.8.0_181-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/626-cluster-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-connector.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
+ echo 'Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java' -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/626-cluster-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-connector.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
+ exec /usr/java/jdk1.8.0_181-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file= -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -cp '/run/cloudera-scm-agent/process/626-cluster-host-inspector:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-connector.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' com.cloudera.cmf.inspector.Inspector input.json output.json DEFAULT
Error: Could not find or load main class com.cloudera.cmf.inspector.Inspector

 

 

 

 

 

 

avatar
Master Guru

@dv_conan  The error is due to missing class form the agent.

Error: Could not find or load main class com.cloudera.cmf.inspector.Inspector

  You have to install again and do a restart. 

yum install cloudera-manager-daemons
service cloudera-scm-agent restart

This should resolve the issue. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
New Contributor

- Make sure in the phase of Install Agent , the heart beat test passed.

- Make sure each host has cm_guid.

[root@redhat1 cloudera-scm-agent]# pwd
/var/lib/cloudera-scm-agent
[root@redhat1 cloudera-scm-agent]# ls
active_parcels.json cm_guid response.avro uuid
[root@redhat1 cloudera-scm-agent]#