Support Questions
Find answers, ask questions, and share your expertise

Failed running performance inspector on cluster Cloudera Manager 6.2.1

Failed running performance inspector on cluster Cloudera Manager 6.2.1

New Contributor

Inspect Network Performance : less /var/log/cloudera-scm-server/cloudera-scm-server.log 2019-12-03 03:48:06,148 INFO scm-web-107:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/v31/clusters/Cluster 1/command s/perfInspector 2019-12-03 03:48:06,206 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ClusterPerfInspector ClusterPerfInspectorCmdArgs{ pingArgs=PerfInspectorPingArgs{pingTimeoutSecs=10, pingCount=10, pingPacketSizeBytes=56}}. 2019-12-03 03:48:06,216 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-12-03 03:48:06,216 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 3 steps in parallel 2019-12-03 03:48:06,217 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Host performance inspector on test-2. ywcloudera.com 2019-12-03 03:48:06,217 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HostPerfInspector HostPerfInspectorCmdArgs{p ingArgs=PerfInspectorPingArgs{pingTimeoutSecs=10, pingCount=10, pingPacketSizeBytes=56}, hosts=[test-3.ywcloudera.com, test-4.ywcloudera.com], bandwidthArgs= PerfInspectorBandwidthArgs{runBandwidthDiagnostics=false, bandwidthTimeoutSecs=10}}. Host: DbHost{id=1, hostId=48769dc9-f49c-4320-a5d3-99f8300b3ffe, hostNam e=test-2.ywcloudera.com} 2019-12-03 03:48:06,218 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-12-03 03:48:06,218 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Running performance inspector on host test-2.ywcloude ra.com. 2019-12-03 03:48:06,233 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Host performance inspector on test-3.ywcloudera.com 2019-12-03 03:48:06,233 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HostPerfInspector HostPerfInspectorCmdArgs{pingArgs=PerfInspectorPingArgs{pingTimeoutSecs=10, pingCount=10, pingPacketSizeBytes=56}, hosts=[test-2.ywcloudera.com, test-4.ywcloudera.com], bandwidthArgs=PerfInspectorBandwidthArgs{runBandwidthDiagnostics=false, bandwidthTimeoutSecs=10}}. Host: DbHost{id=3, hostId=bba21141-fdd5-4323-a7c4-4cfd646b7719, hostName=test-3.ywcloudera.com} 2019-12-03 03:48:06,233 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-12-03 03:48:06,234 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Running performance inspector on host test-3.ywcloudera.com. 2019-12-03 03:48:06,235 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Host performance inspector on test-4.ywcloudera.com 2019-12-03 03:48:06,235 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HostPerfInspector HostPerfInspectorCmdArgs{pingArgs=PerfInspectorPingArgs{pingTimeoutSecs=10, pingCount=10, pingPacketSizeBytes=56}, hosts=[test-2.ywcloudera.com, test-3.ywcloudera.com], bandwidthArgs=PerfInspectorBandwidthArgs{runBandwidthDiagnostics=false, bandwidthTimeoutSecs=10}}. Host: DbHost{id=2, hostId=a2f23653-86c7-49a1-a6a1-10819d171a95, hostName=test-4.ywcloudera.com} 2019-12-03 03:48:06,236 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-12-03 03:48:06,236 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Running performance inspector on host test-4.ywcloudera.com. 2019-12-03 03:48:06,432 INFO scm-web-107:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/v31/clusters/Cluster 1/commands/perfInspector, Status:200 2019-12-03 03:48:06,475 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=10 name=null host=48769dc9-f49c-4320-a5d3-99f8300b3ffe/test-2.ywcloudera.com 2019-12-03 03:48:06,478 WARN avro-servlet-hb-processor-1:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=12 name=null host=a2f23653-86c7-49a1-a6a1-10819d171a95/test-4.ywcloudera.com 2019-12-03 03:48:06,508 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=11 name=null host=bba21141-fdd5-4323-a7c4-4cfd646b7719/test-3.ywcloudera.com 2019-12-03 03:48:08,756 ERROR CommandPusher:com.cloudera.cmf.command.HostPerfInspectorCommand: Either command 42 or fetching result file from host test-2.ywcloudera.com failed 2019-12-03 03:48:08,756 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 42(HostPerfInspector) has completed. finalstate:FINISHED, success:false, msg:Failed running performance inspector on host test-2.ywcloudera.com. 2019-12-03 03:48:08,759 ERROR CommandPusher:com.cloudera.cmf.command.HostPerfInspectorCommand: Either command 44 or fetching result file from host test-4.ywcloudera.com failed 2019-12-03 03:48:08,759 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 44(HostPerfInspector) has completed. finalstate:FINISHED, success:false, msg:Failed running performance inspector on host test-4.ywcloudera.com. 2019-12-03 03:48:08,759 ERROR CommandPusher:com.cloudera.cmf.command.HostPerfInspectorCommand: Either command 43 or fetching result file from host test-3.ywcloudera.com failed 2019-12-03 03:48:08,759 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 43(HostPerfInspector) has completed. finalstate:FINISHED, success:false, msg:Failed running performance inspector on host test-3.ywcloudera.com. 2019-12-03 03:48:08,761 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 41(ClusterPerfInspector) has completed. finalstate:FINISHED, success:false, msg:Failed running performance inspector on cluster Cluster 1. less /var/log/cloudera-scm-agent/cloudera-scm-agent.log [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO [13-host-perf-inspector] Instantiating process [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO [13-host-perf-inspector] Updating process: True {} [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO First time to activate the process [13-host-perf-inspector]. [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Created /var/run/cloudera-scm-agent/process/13-host-perf-inspector [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Chowning /var/run/cloudera-scm-agent/process/13-host-perf-inspector to cloudera-scm (996) cloudera-scm (994) [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/13-host-perf-inspector to 0751 [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Created /var/run/cloudera-scm-agent/process/13-host-perf-inspector/logs [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Chowning /var/run/cloudera-scm-agent/process/13-host-perf-inspector/logs to cloudera-scm (996) cloudera-scm (994) [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/13-host-perf-inspector/logs to 0751 [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO [13-host-perf-inspector] Refreshing process files: None [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 parcel INFO prepare_environment begin: {}, [], [] [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 parcel INFO No parcels activated for use [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 __init__ INFO Extracted 1 files and 0 dirs to /var/run/cloudera-scm-agent/process/13-host-perf-inspector. [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 process INFO [13-host-perf-inspector] Launching process. one-off True, command perf/host_perf_diag.py, args [u'input.json', u'logs/result.json'] [03/Dec/2019 03:48:06 +0000] 7369 Thread-14 supervisor WARNING Failed while getting process info. Retrying. () [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 supervisor INFO Triggering supervisord update. [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 util INFO Using generic audit plugin for process host-perf-inspector [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 util INFO Creating metadata plugin for process host-perf-inspector [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 util INFO Using specific metadata plugin for process host-perf-inspector [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 util INFO Using generic metadata plugin for process host-perf-inspector [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 process INFO Begin audit plugin refresh [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 throttling_logger INFO (5 skipped) Scheduling a refresh for Audit Plugin for host-perf-inspector with count 1 pipelines names ['']. [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 process INFO Begin metadata plugin refresh [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 process INFO Not creating a monitor for 13-host-perf-inspector: should_monitor returns false [03/Dec/2019 03:48:08 +0000] 7369 Thread-14 process INFO Daemon refresh complete for process 13-host-perf-inspector. [03/Dec/2019 03:48:08 +0000] 7369 Thread-15 process INFO [13-host-perf-inspector] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)} [03/Dec/2019 03:48:08 +0000] 7369 Thread-15 process INFO [13-host-perf-inspector] Deactivating process [03/Dec/2019 03:48:09 +0000] 7369 Thread-15 process INFO [13-host-perf-inspector] Unregistered supervisor process STOPPED [03/Dec/2019 03:48:09 +0000] 7369 Audit-Plugin navigator_plugin INFO Pipelines updated for Audit Plugin: [] [03/Dec/2019 03:48:09 +0000] 7369 Audit-Plugin throttling_logger INFO (3 skipped) Refreshing Audit Plugin for host-perf-inspector with count 0 pipelines names []. [03/Dec/2019 03:48:09 +0000] 7369 Metadata-Plugin navigator_plugin INFO Pipelines updated for Metadata Plugin: [] [03/Dec/2019 03:48:11 +0000] 7369 Thread-15 supervisor INFO Triggering supervisord update. [03/Dec/2019 03:48:11 +0000] 7369 Thread-15 process INFO [13-host-perf-inspector] stopping monitors [03/Dec/2019 03:48:14 +0000] 7369 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for host-perf-inspector with count 0 pipelines names []. [03/Dec/2019 03:48:14 +0000] 7369 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for host-perf-inspector with count 0 pipelines names []. environment [root@test-4 ~]# uname -a Linux test-4.ywcloudera.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux [root@test-4 ~]# hostname test-4.ywcloudera.com [root@test-4 ~]# getenforce

 

Permissive firewalld has stop where is the problem ?

3 REPLIES 3

Re: Failed running performance inspector on cluster Cloudera Manager 6.2.1

Cloudera Employee

@wym Could you please check if the host is heart beating or not and also check if there is any duplicate hostname  In Cloudera manager go to CM==> Hosts==> All Hosts page.

Please try to run with the API command and see if the works. A link has mentioned below.

https://docs.cloudera.com/documentation/enterprise/6/6.2/topics/cm_network_perf_inspector.html#conce...

Re: Failed running performance inspector on cluster Cloudera Manager 6.2.1

New Contributor
Thank you for your help "GET /heartbeat HTTP/1.1" 200 2 "" "AHC/1.0" there is no duplicate hostname I don't know how to run with the API Add some phenomenon: add hdfs,hive,impala,kudu,zk success , but failed when Enable hdfs High Availability , error log

Re: Failed running performance inspector on cluster Cloudera Manager 6.2.1

New Contributor
less /var/log/cloudera-scm-server/cloudera-scm-server.log 2019-12-05 02:30:08,424 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=95ms, min=64ms, max=279ms. 2019-12-05 02:31:11,532 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (13 skipped) AgentAvroServlet: heartbeat processing stats: average=95ms, min=64ms, max=279ms. 2019-12-05 02:31:33,587 INFO scm-web-2610:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/services/8/enableHA/command 2019-12-05 02:31:33,782 INFO scm-web-2610:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command EnableNNHA EnableNNHACmdArgs{targetRoles=[], args=[]}. Service: DbService{id=8, name=hdfs} 2019-12-05 02:31:33,898 INFO scm-web-2610:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 20 steps in sequence 2019-12-05 02:31:33,898 INFO scm-web-2610:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Validates that specified directories exist on host test-3.ywcloudera.com, are writable, and are empty. Can optionally clear directories. 2019-12-05 02:31:34,086 INFO scm-web-2610:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/services/8/enableHA/command, Status:200 2019-12-05 02:31:34,297 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=61 name=null host=998adde1-4403-42bd-abb1-e3d1efd4cbe9/test-3.ywcloudera.com 2019-12-05 02:32:23,439 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (14 skipped) AgentAvroServlet: heartbeat processing stats: average=96ms, min=64ms, max=279ms. less /var/log/cloudera-scm-agent/cloudera-scm-agent.log [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Instantiating process [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Updating process: True {} [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO First time to activate the process [62-host-validate-writable-empty-dirs]. [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Created /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chowning /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs to hdfs (994) hdfs (991) [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs to 0751 [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Created /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chowning /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs to hdfs (994) hdfs (991) [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs to 0751 [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Refreshing process files: None [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 parcel INFO prepare_environment begin: {}, [], [] [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 parcel INFO No parcels activated for use [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 __init__ INFO Extracted 0 files and 0 dirs to /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs. [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Launching process. one-off True, command common/validate_writable_empty_dirs.sh, args [u'true', u'/home/hadoop/data_cm/dfs/nn'] [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 supervisor WARNING Failed while getting process info. Retrying. () [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 supervisor INFO Triggering supervisord update. [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using generic audit plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Creating metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using specific metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using generic metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Begin audit plugin refresh [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Begin metadata plugin refresh [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Not creating a monitor for 62-host-validate-writable-empty-dirs: should_monitor returns false [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Daemon refresh complete for process 62-host-validate-writable-empty-dirs. [05/Dec/2019 02:31:37 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)} [05/Dec/2019 02:31:37 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Deactivating process [05/Dec/2019 02:31:37 +0000] 30663 Metadata-Plugin navigator_plugin INFO Pipelines updated for Metadata Plugin: [] [05/Dec/2019 02:31:37 +0000] 30663 Audit-Plugin navigator_plugin INFO Pipelines updated for Audit Plugin: [] [05/Dec/2019 02:31:38 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Unregistered supervisor process STOPPED [05/Dec/2019 02:31:40 +0000] 30663 Thread-17 supervisor INFO Triggering supervisord update. [05/Dec/2019 02:31:40 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] stopping monitors [05/Dec/2019 02:31:42 +0000] 30663 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for host-validate-writable-empty-dirs with count 0 pipelines names []. [05/Dec/2019 02:31:42 +0000] 30663 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for host-validate-writable-empty-dirs with count 0 pipelines names [].