less /var/log/cloudera-scm-server/cloudera-scm-server.log 2019-12-05 02:30:08,424 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (11 skipped) AgentAvroServlet: heartbeat processing stats: average=95ms, min=64ms, max=279ms. 2019-12-05 02:31:11,532 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (13 skipped) AgentAvroServlet: heartbeat processing stats: average=95ms, min=64ms, max=279ms. 2019-12-05 02:31:33,587 INFO scm-web-2610:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/services/8/enableHA/command 2019-12-05 02:31:33,782 INFO scm-web-2610:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command EnableNNHA EnableNNHACmdArgs{targetRoles=[], args=[]}. Service: DbService{id=8, name=hdfs} 2019-12-05 02:31:33,898 INFO scm-web-2610:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 20 steps in sequence 2019-12-05 02:31:33,898 INFO scm-web-2610:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Validates that specified directories exist on host test-3.ywcloudera.com, are writable, and are empty. Can optionally clear directories. 2019-12-05 02:31:34,086 INFO scm-web-2610:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/services/8/enableHA/command, Status:200 2019-12-05 02:31:34,297 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=61 name=null host=998adde1-4403-42bd-abb1-e3d1efd4cbe9/test-3.ywcloudera.com 2019-12-05 02:32:23,439 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (14 skipped) AgentAvroServlet: heartbeat processing stats: average=96ms, min=64ms, max=279ms. less /var/log/cloudera-scm-agent/cloudera-scm-agent.log [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Instantiating process [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Updating process: True {} [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO First time to activate the process [62-host-validate-writable-empty-dirs]. [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Created /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chowning /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs to hdfs (994) hdfs (991) [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs to 0751 [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Created /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chowning /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs to hdfs (994) hdfs (991) [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs/logs to 0751 [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Refreshing process files: None [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 parcel INFO prepare_environment begin: {}, [], [] [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 parcel INFO No parcels activated for use [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 __init__ INFO Extracted 0 files and 0 dirs to /var/run/cloudera-scm-agent/process/62-host-validate-writable-empty-dirs. [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Launching process. one-off True, command common/validate_writable_empty_dirs.sh, args [u'true', u'/home/hadoop/data_cm/dfs/nn'] [05/Dec/2019 02:31:34 +0000] 30663 Thread-17 supervisor WARNING Failed while getting process info. Retrying. () [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 supervisor INFO Triggering supervisord update. [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using generic audit plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Creating metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using specific metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 util INFO Using generic metadata plugin for process host-validate-writable-empty-dirs [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Begin audit plugin refresh [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Begin metadata plugin refresh [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Not creating a monitor for 62-host-validate-writable-empty-dirs: should_monitor returns false [05/Dec/2019 02:31:36 +0000] 30663 Thread-17 process INFO Daemon refresh complete for process 62-host-validate-writable-empty-dirs. [05/Dec/2019 02:31:37 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)} [05/Dec/2019 02:31:37 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Deactivating process [05/Dec/2019 02:31:37 +0000] 30663 Metadata-Plugin navigator_plugin INFO Pipelines updated for Metadata Plugin: [] [05/Dec/2019 02:31:37 +0000] 30663 Audit-Plugin navigator_plugin INFO Pipelines updated for Audit Plugin: [] [05/Dec/2019 02:31:38 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] Unregistered supervisor process STOPPED [05/Dec/2019 02:31:40 +0000] 30663 Thread-17 supervisor INFO Triggering supervisord update. [05/Dec/2019 02:31:40 +0000] 30663 Thread-17 process INFO [62-host-validate-writable-empty-dirs] stopping monitors [05/Dec/2019 02:31:42 +0000] 30663 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for host-validate-writable-empty-dirs with count 0 pipelines names []. [05/Dec/2019 02:31:42 +0000] 30663 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for host-validate-writable-empty-dirs with count 0 pipelines names []. ... View more