Failed running performance inspector on cluster Cloudera Manager 6.2.1

[#1]

Inspect Network Performance : less /var/log/cloudera-scm-server/cloudera-scm-server.log 2019-12-03 03:48:06,148 INFO scm-web-107:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/v31/clusters/Cluster 1/command s/perfInspector 2019-12-03 03:48:06,206 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ClusterPerfInspector ClusterPerfInspectorCmdArgs{ pingArgs=PerfInspectorPingArgs{pingTimeoutSecs=10, pingCount=10, pingPacketSizeBytes=56}}. 2019-12-03 03:48:06,216 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-12-03 03:48:06,216 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 3 steps in parallel 2019-12-03 03:48:06,217 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Host performance inspector on test-2. ywcloudera.com 2019-12-03 03:48:06,217 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HostPerfInspector HostPerfInspectorCmdArgs{p ingArgs=PerfInspectorPingArgs{pingTimeoutSecs=10, pingCount=10, pingPacketSizeBytes=56}, hosts=[test-3.ywcloudera.com, test-4.ywcloudera.com], bandwidthArgs= PerfInspectorBandwidthArgs{runBandwidthDiagnostics=false, bandwidthTimeoutSecs=10}}. Host: DbHost{id=1, hostId=48769dc9-f49c-4320-a5d3-99f8300b3ffe, hostNam e=test-2.ywcloudera.com} 2019-12-03 03:48:06,218 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-12-03 03:48:06,218 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Running performance inspector on host test-2.ywcloude ra.com. 2019-12-03 03:48:06,233 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Host performance inspector on test-3.ywcloudera.com 2019-12-03 03:48:06,233 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HostPerfInspector HostPerfInspectorCmdArgs{pingArgs=PerfInspectorPingArgs{pingTimeoutSecs=10, pingCount=10, pingPacketSizeBytes=56}, hosts=[test-2.ywcloudera.com, test-4.ywcloudera.com], bandwidthArgs=PerfInspectorBandwidthArgs{runBandwidthDiagnostics=false, bandwidthTimeoutSecs=10}}. Host: DbHost{id=3, hostId=bba21141-fdd5-4323-a7c4-4cfd646b7719, hostName=test-3.ywcloudera.com} 2019-12-03 03:48:06,233 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-12-03 03:48:06,234 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Running performance inspector on host test-3.ywcloudera.com. 2019-12-03 03:48:06,235 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Host performance inspector on test-4.ywcloudera.com 2019-12-03 03:48:06,235 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HostPerfInspector HostPerfInspectorCmdArgs{pingArgs=PerfInspectorPingArgs{pingTimeoutSecs=10, pingCount=10, pingPacketSizeBytes=56}, hosts=[test-2.ywcloudera.com, test-3.ywcloudera.com], bandwidthArgs=PerfInspectorBandwidthArgs{runBandwidthDiagnostics=false, bandwidthTimeoutSecs=10}}. Host: DbHost{id=2, hostId=a2f23653-86c7-49a1-a6a1-10819d171a95, hostName=test-4.ywcloudera.com} 2019-12-03 03:48:06,236 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-12-03 03:48:06,236 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Running performance inspector on host test-4.ywcloudera.com. 2019-12-03 03:48:06,432 INFO scm-web-107:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/v31/clusters/Cluster 1/commands/perfInspector, Status:200 2019-12-03 03:48:06,475 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=10 name=null host=48769dc9-f49c-4320-a5d3-99f8300b3ffe/test-2.ywcloudera.com 2019-12-03 03:48:06,478 WARN avro-servlet-hb-processor-1:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=12 name=null host=a2f23653-86c7-49a1-a6a1-10819d171a95/test-4.ywcloudera.com 2019-12-03 03:48:06,508 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=11 name=null host=bba21141-fdd5-4323-a7c4-4cfd646b7719/test-3.ywcloudera.com 2019-12-03 03:48:08,756 ERROR CommandPusher:com.cloudera.cmf.command.HostPerfInspectorCommand: Either command 42 or fetching result file from host test-2.ywcloudera.com failed 2019-12-03 03:48:08,756 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 42(HostPerfInspector) has completed. finalstate:FINISHED, success:false, msg:Failed running performance inspector on host test-2.ywcloudera.com. 2019-12-03 03:48:08,759 ERROR CommandPusher:com.cloudera.cmf.command.HostPerfInspectorCommand: Either command 44 or fetching result file from host test-4.ywcloudera.com failed 2019-12-03 03:48:08,759 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 44(HostPerfInspector) has completed. finalstate:FINISHED, success:false, msg:Failed running performance inspector on host test-4.ywcloudera.com. 2019-12-03 03:48:08,759 ERROR CommandPusher:com.cloudera.cmf.command.HostPerfInspectorCommand: Either command 43 or fetching result file from host test-3.ywcloudera.com failed 2019-12-03 03:48:08,759 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 43(HostPerfInspector) has completed. finalstate:FINISHED, success:false, msg:Failed running performance inspector on host test-3.ywcloudera.com. 2019-12-03 03:48:08,761 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 41(ClusterPerfInspector) has completed. finalstate:FINISHED, success:false, msg:Failed running performance inspector on cluster Cluster 1. less /var/log/cloudera-scm-agent/cloudera-scm-agent.log [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO [13-host-perf-inspector] Instantiating process [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO [13-host-perf-inspector] Updating process: True {} [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO First time to activate the process [13-host-perf-inspector]. [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Created /var/run/cloudera-scm-agent/process/13-host-perf-inspector [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Chowning /var/run/cloudera-scm-agent/process/13-host-perf-inspector to cloudera-scm (996) cloudera-scm (994) [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/13-host-perf-inspector to 0751 [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Created /var/run/cloudera-scm-agent/process/13-host-perf-inspector/logs [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Chowning /var/run/cloudera-scm-agent/process/13-host-perf-inspector/logs to cloudera-scm (996) cloudera-scm (994) [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/13-host-perf-inspector/logs to 0751 [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO [13-host-perf-inspector] Refreshing process files: None [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 parcel INFO prepare_environment begin: {}, [], [] [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 parcel INFO No parcels activated for use [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 __init__ INFO Extracted 1 files and 0 dirs to /var/run/cloudera-scm-agent/process/13-host-perf-inspector. [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO [13-host-perf-inspector] Launching process. one-off True, command perf/host_perf_diag.py, args [u'input.json', u'logs/result.json'] [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 supervisor WARNING Failed while getting process info. Retrying. () [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 supervisor INFO Triggering supervisord update. [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 util INFO Using generic audit plugin for process host-perf-inspector [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 util INFO Creating metadata plugin for process host-perf-inspector [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 util INFO Using specific metadata plugin for process host-perf-inspector [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 util INFO Using generic metadata plugin for process host-perf-inspector [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 process INFO Begin audit plugin refresh [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 throttling_logger INFO (5 skipped) Scheduling a refresh for Audit Plugin for host-perf-inspector with count 1 pipelines names ['']. [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 process INFO Begin metadata plugin refresh [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 process INFO Not creating a monitor for 13-host-perf-inspector: should_monitor returns false [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 process INFO Daemon refresh complete for process 13-host-perf-inspector. [03/Dec/2019 03:48:08 +0000] 7369 Thread-15 process INFO [13-host-perf-inspector] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)} [03/Dec/2019 03:48:08 +0000] 7369 Thread-15 process INFO [13-host-perf-inspector] Deactivating process [03/Dec/2019 03:48:09 +0000] 7369 Thread-15 process INFO [13-host-perf-inspector] Unregistered supervisor process STOPPED [03/Dec/2019 03:48:09 +0000] 7369 Audit-Plugin navigator_plugin INFO Pipelines updated for Audit Plugin: [] [03/Dec/2019 03:48:09 +0000] 7369 Audit-Plugin throttling_logger INFO (3 skipped) Refreshing Audit Plugin for host-perf-inspector with count 0 pipelines names []. [03/Dec/2019 03:48:09 +0000] 7369 Metadata-Plugin navigator_plugin INFO Pipelines updated for Metadata Plugin: [] [03/Dec/2019 03:48:11 +0000] 7369 Thread-15 supervisor INFO Triggering supervisord update. [03/Dec/2019 03:48:11 +0000] 7369 Thread-15 process INFO [13-host-perf-inspector] stopping monitors [03/Dec/2019 03:48:14 +0000] 7369 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for host-perf-inspector with count 0 pipelines names []. [03/Dec/2019 03:48:14 +0000] 7369 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for host-perf-inspector with count 0 pipelines names []. environment [root@test-4 ~]# uname -a Linux test-4.ywcloudera.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux [root@test-4 ~]# hostname test-4.ywcloudera.com [root@test-4 ~]# getenforce

Permissive firewalld has stop where is the problem ?

[#2]

@wym Could you please check if the host is heart beating or not and also check if there is any duplicate hostname  In Cloudera manager go to CM==> Hosts==> All Hosts page.

Please try to run with the API command and see if the works. A link has mentioned below.

https://docs.cloudera.com/documentation/enterprise/6/6.2/topics/cm_network_perf_inspector.html#conce...

[#3]

Thank you for your help "GET /heartbeat HTTP/1.1" 200 2 "" "AHC/1.0" there is no duplicate hostname I don't know how to run with the API Add some phenomenon: add hdfs,hive,impala,kudu,zk success , but failed when Enable hdfs High Availability , error log

[#4]

less /var/log/cloudera-scm-server/cloudera-scm-server.log 2019-12-05 02:30:08,424 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=95ms, min=64ms, max=279ms. 2019-12-05 02:31:11,532 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (13 skipped) AgentAvroServlet: heartbeat processing stats: average=95ms, min=64ms, max=279ms. 2019-12-05 02:31:33,587 INFO scm-web-2610:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/services/8/enableHA/command 2019-12-05 02:31:33,782 INFO scm-web-2610:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command EnableNNHA EnableNNHACmdArgs{targetRoles=[], args=[]}. Service: DbService{id=8, name=hdfs} 2019-12-05 02:31:33,898 INFO scm-web-2610:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 20 steps in sequence 2019-12-05 02:31:33,898 INFO scm-web-2610:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Validates that specified directories exist on host test-3.ywcloudera.com, are writable, and are empty. Can optionally clear directories. 2019-12-05 02:31:34,086 INFO scm-web-2610:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/services/8/enableHA/command, Status:200 2019-12-05 02:31:34,297 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=61 name=null host=998adde1-4403-42bd-abb1-e3d1efd4cbe9/test-3.ywcloudera.com 2019-12-05 02:32:23,439 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (14 skipped) AgentAvroServlet: heartbeat processing stats: average=96ms, min=64ms, max=279ms. less /var/log/cloudera-scm-agent/cloudera-scm-agent.log [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Instantiating process [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Updating process: True {} [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO First time to activate the process [62-host-validate-writable-empty-dirs]. [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Created /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chowning /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs to hdfs (994) hdfs (991) [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs to 0751 [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Created /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chowning /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs to hdfs (994) hdfs (991) [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs to 0751 [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Refreshing process files: None [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 parcel INFO prepare_environment begin: {}, [], [] [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 parcel INFO No parcels activated for use [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 __init__ INFO Extracted 0 files and 0 dirs to /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs. [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Launching process. one-off True, command common/validate_writable_empty_dirs.sh, args [u'true', u'/home/hadoop/data_cm/dfs/nn'] [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 supervisor WARNING Failed while getting process info. Retrying. () [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 supervisor INFO Triggering supervisord update. [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using generic audit plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Creating metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using specific metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using generic metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Begin audit plugin refresh [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Begin metadata plugin refresh [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Not creating a monitor for 62-host-validate-writable-empty-dirs: should_monitor returns false [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Daemon refresh complete for process 62-host-validate-writable-empty-dirs. [05/Dec/2019 02:31:37 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)} [05/Dec/2019 02:31:37 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Deactivating process [05/Dec/2019 02:31:37 +0000] 30663 Metadata-Plugin navigator_plugin INFO Pipelines updated for Metadata Plugin: [] [05/Dec/2019 02:31:37 +0000] 30663 Audit-Plugin navigator_plugin INFO Pipelines updated for Audit Plugin: [] [05/Dec/2019 02:31:38 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Unregistered supervisor process STOPPED [05/Dec/2019 02:31:40 +0000] 30663 Thread-17 supervisor INFO Triggering supervisord update. [05/Dec/2019 02:31:40 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] stopping monitors [05/Dec/2019 02:31:42 +0000] 30663 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for host-validate-writable-empty-dirs with count 0 pipelines names []. [05/Dec/2019 02:31:42 +0000] 30663 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for host-validate-writable-empty-dirs with count 0 pipelines names [].