4545 collection of host statistics data reports error room overflow, server memory usage ratio is too high, resulting in server pump machine

[#1]

[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 process INFO [4545-collect-host-statistics] Instantiating process
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 process INFO [4545-collect-host-statistics] Updating process: True {}
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 process INFO First time to activate the process [4545-collect-host-statistics].
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Created /var/run/cloudera-scm-agent/process/4545-collect-host-statistics
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Chowning /var/run/cloudera-scm-agent/process/4545-collect-host-statistics to root (0) root (0)
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/4545-collect-host-statistics to 0751
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Created /var/run/cloudera-scm-agent/process/4545-collect-host-statistics/logs
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Chowning /var/run/cloudera-scm-agent/process/4545-collect-host-statistics/logs to root (0) root (0)
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/4545-collect-host-statistics/logs to 0751
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 process INFO [4545-collect-host-statistics] Refreshing process files: None
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 parcel INFO prepare_environment begin: {}, [], []
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 parcel INFO No parcels activated for use
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 __init__ INFO Extracted 1 files and 0 dirs to /var/run/cloudera-scm-agent/process/4545-collect-host-statistics.
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 process INFO [4545-collect-host-statistics] Launching process. one-off True, command support/collect_host_stats.sh, args []
[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 supervisor WARNING Failed while getting process info. Retrying. (<Fault 10: 'BAD_NAME: 4545-collect-host-statistics'>)
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 supervisor INFO Triggering supervisord update.
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 util INFO Using generic audit plugin for process collect-host-statistics
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 util INFO Creating metadata plugin for process collect-host-statistics
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 util INFO Using specific metadata plugin for process collect-host-statistics
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 util INFO Using generic metadata plugin for process collect-host-statistics
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 process INFO Begin audit plugin refresh
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 throttling_logger INFO (3 skipped) Scheduling a refresh for Audit Plugin for collect-host-statistics with count 1 pipelines names [''].
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 process INFO Begin metadata plugin refresh
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 process INFO Not creating a monitor for 4545-collect-host-statistics: should_monitor returns false
[22/Jun/2025 12:40:04 +0000] 16316 Thread-15 process INFO Daemon refresh complete for process 4545-collect-host-statistics.
[22/Jun/2025 12:40:05 +0000] 16316 Audit-Plugin navigator_plugin INFO Pipelines updated for Audit Plugin: []
[22/Jun/2025 12:40:05 +0000] 16316 Audit-Plugin throttling_logger INFO (3 skipped) Refreshing Audit Plugin for collect-host-statistics with count 0 pipelines names [].
[22/Jun/2025 12:40:05 +0000] 16316 Metadata-Plugin navigator_plugin INFO Pipelines updated for Metadata Plugin: []

[#2]

Hello @Jecky 

Thank you for reaching out to cloudera community

Could you please confirm the CM and CDP versions here?

Also, could you please elaborate on the actual problem? Are you facing issues while collecting the diagnostic bundle?

Thank you
Kshitij Upadhyay

[#3]

Hello @upadhyayk04 

cm version is 6.3.1

In actual use, there is a server that often does not release memory after executing the lsof - n - P command to be collected, until the server crashes. The problem has not been found on other servers, and there are no abnormal tasks on the abnormal server. The logs show that only normal collection is being done, and there are no other obvious errors

thank you

[#4]

Hello @Jecky 

On the abnormal server, do you have anything suspicious in /var/log/messages or /var/log/cloudera-scm-agent/cloudera-scm-agent.log

[#5]

Hello @upadhyayk04 

When I checked the local logs, I found some errors in the/var/log/cloudera scm agent/status standard output. log. I don't know if it will have an impact

 - - [22/Jun/2025:13:41:24] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fstacks%2Fhadoop-cmf-yarn-NODEMANAGER-olap-rtc-2.online.uc-bj-2b..log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1940 "" "Java/1.8.0_162"

('/var/log/cloudera-scm-agent/cloudera-flood.log', 29262586, 524118)

('/var/log/cloudera-scm-agent/status-stderr.log', 28738468, 1698)('/var/log/cloudera-scm-agent/supervisord.log', 28736770 - - [22/Jun/2025:13:41:24] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fstacks%2Fhadoop-cmf-hdfs-DATANODE-olap-rtc-2.online.uc-bj-2b..log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1934 "" "Java/1.8.0_162"

, 207506)

('/var/log/cloudera-scm-agent/scm-status.log', 28529264, 3726816)

('/var/log/cloudera-scm-agent/supervisord.out', 24802448, 207915)

('/var/log/cloudera-scm-agent/cmf_listener.log', 24594533, 205)

('/var/log/cloudera-scm-agent/status-stdout.log', 24594328, 2438122)

('/var/log/cloudera-scm-agent/cloudera-scm-agent.log', 22156206, 189407)

('/var/log/cloudera-scm-agent/cloudera-flood.log.1', 21966799, 1048496)

('/var/log/cloudera-scm-agent/cloudera-scm-agent.log.1', 20918303, 10485605)

('/var/log/cloudera-scm-agent/cloudera-scm-agent.log.2', 10432698, 10485743)

('/var/log/cloudera-scm-agent/cloudera-flood.log.2', 0, 1048508)

('/var/log/cloudera-scm-agent/cloudera-flood.log.3', 0, 1048557)

('/var/log/cloudera-scm-agent/cloudera-scm-agent.log.3', 0, 10485664)

('/var/log/cloudera-scm-agent/cloudera-flood.log.4', 0, 1048474)

('/var/log/cloudera-scm-agent/cloudera-scm-agent.log.4', 0, 10485582)

('/var/log/cloudera-scm-agent/cloudera-flood.log.5', 0, 1048439)

('/var/log/cloudera-scm-agent/cloudera-scm-agent.log.5', 0, 10485577)

 - - [22/Jun/2025:13:41:24] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-httpfs%2Fstacks%2Fhadoop-cmf-hdfs-HTTPFS-olap-rtc-2.online.uc-bj-2b..log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1934 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:24] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fstacks%2Fhadoop-cmf-hdfs-FAILOVERCONTROLLER-olap-rtc-2.online.uc-bj-2b..log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1954 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:24] "GET /process/4329-yarn-RESOURCEMANAGER/files/logs/stdout.log HTTP/1.1" 206 482 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:24] "GET /process/4550-host-inspector/files/logs/stdout.log HTTP/1.1" 206 6878 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:24] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fstacks%2Fhadoop-cmf-hdfs-NAMENODE-olap-rtc-2.online.uc-bj-2b..log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1934 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:24] "GET /process/355-zookeeper-server/files/logs/stdout.log HTTP/1.1" 206 262144 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:24] "GET /process/4329-yarn-RESOURCEMANAGER/files/logs/stderr.log HTTP/1.1" 200 24687 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:24] "GET /process/4550-host-inspector/files/logs/stderr.log HTTP/1.1" 200 7256 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:25] "GET /process/355-zookeeper-server/files/logs/stderr.log HTTP/1.1" 200 20521 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:25] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1750567200011&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fhadoop-cmf-yarn-NODEMANAGER-olap-rtc-2.online.uc-bj-2b..log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:26] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.yarn_-7702462360475554876/files/logs/stdout.log HTTP/1.1" 206 561 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:26] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.yarn_-7702462360475554876/files/logs/stderr.log HTTP/1.1" 200 20096 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:26] "GET /process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1684164223045907750/files/logs/stdout.log HTTP/1.1" 206 1209 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:26] "GET /process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1684164223045907750/files/logs/stderr.log HTTP/1.1" 200 340011 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:27] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_-8215012917422850757/files/logs/stdout.log HTTP/1.1" 206 563 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:27] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_-8215012917422850757/files/logs/stderr.log HTTP/1.1" 200 18892 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /process/4545-collect-host-statistics/files/logs/stdout.log HTTP/1.1" 206 14664 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /download_log?path=%2Fvar%2Flog%2Fzookeeper%2Fstacks%2Fzookeeper-cmf-zookeeper-SERVER-olap-rtc-2.online.uc-bj-2b..log.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1924 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /process/4545-collect-host-statistics/files/logs/stderr.log HTTP/1.1" 200 26279 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /process/4209-yarn-NODEMANAGER/files/logs/stdout.log HTTP/1.1" 206 474 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /process/2704-hdfs-DATANODE/files/logs/stdout.log HTTP/1.1" 206 331 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /process/2693-hdfs-FAILOVERCONTROLLER/files/logs/stdout.log HTTP/1.1" 206 353 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /process/4209-yarn-NODEMANAGER/files/logs/stderr.log HTTP/1.1" 200 24954 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /process/2704-hdfs-DATANODE/files/logs/stderr.log HTTP/1.1" 200 22914 "" "Java/1.8.0_162"

 - - [22/Jun/2025:13:41:28] "GET /process/2693-hdfs-FAILOVERCONTROLLER/files/logs/stderr.log HTTP/1.1" 200 17982 "" "Java/1.8.0_162"

[#6]

Hello @upadhyayk04 

The suspicious content is as follows

[22/Jun/2025 13:40:38 +0000] 16316 Thread-14 process INFO [4545-collect-host-statistics] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[22/Jun/2025 13:40:38 +0000] 16316 Thread-14 process INFO [4545-collect-host-statistics] Deactivating process
[22/Jun/2025 13:40:39 +0000] 16316 Thread-14 process INFO [4545-collect-host-statistics] Unregistered supervisor process STOPPED
[22/Jun/2025 13:40:41 +0000] 16316 Thread-14 supervisor INFO Triggering supervisord update.
[22/Jun/2025 13:40:41 +0000] 16316 Thread-14 process INFO [4545-collect-host-statistics] stopping monitors
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 process INFO [4550-host-inspector] Instantiating process
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 process INFO [4550-host-inspector] Updating process: True {}
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 process INFO First time to activate the process [4550-host-inspector].
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 agent INFO Created /var/run/cloudera-scm-agent/process/4550-host-inspector
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 agent INFO Chowning /var/run/cloudera-scm-agent/process/4550-host-inspector to root (0) root (0)
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/4550-host-inspector to 0751
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 agent INFO Created /var/run/cloudera-scm-agent/process/4550-host-inspector/logs
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 agent INFO Chowning /var/run/cloudera-scm-agent/process/4550-host-inspector/logs to root (0) root (0)
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/4550-host-inspector/logs to 0751
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 process INFO [4550-host-inspector] Refreshing process files: None
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 parcel INFO prepare_environment begin: {}, [], []
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 parcel INFO No parcels activated for use
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 __init__ INFO Extracted 1 files and 0 dirs to /var/run/cloudera-scm-agent/process/4550-host-inspector.
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 process INFO [4550-host-inspector] Launching process. one-off True, command mgmt/mgmt.sh, args [u'inspector', u'input.json', u'output.json', u'ALL']
[22/Jun/2025 13:40:41 +0000] 16316 Thread-17 supervisor WARNING Failed while getting process info. Retrying. (<Fault 10: 'BAD_NAME: 4550-host-inspector'>)
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 supervisor INFO Triggering supervisord update.
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 util INFO Using generic audit plugin for process host-inspector
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 util INFO Creating metadata plugin for process host-inspector
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 util INFO Using specific metadata plugin for process host-inspector
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 util INFO Using generic metadata plugin for process host-inspector
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 process INFO Begin audit plugin refresh
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 throttling_logger INFO (1 skipped) Scheduling a refresh for Audit Plugin for host-inspector with count 1 pipelines names [''].
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 process INFO Begin metadata plugin refresh
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 process INFO Not creating a monitor for 4550-host-inspector: should_monitor returns false
[22/Jun/2025 13:40:43 +0000] 16316 Thread-17 process INFO Daemon refresh complete for process 4550-host-inspector.
[22/Jun/2025 13:40:43 +0000] 16316 MainThread agent WARNING Long HB processing time: 5.26821017265
[22/Jun/2025 13:40:43 +0000] 16316 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for collect-host-statistics with count 0 pipelines names [].
[22/Jun/2025 13:40:43 +0000] 16316 Audit-Plugin navigator_plugin INFO Pipelines updated for Audit Plugin: []
[22/Jun/2025 13:40:43 +0000] 16316 Audit-Plugin throttling_logger INFO (1 skipped) Refreshing Audit Plugin for host-inspector with count 0 pipelines names [].
[22/Jun/2025 13:40:44 +0000] 16316 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for collect-host-statistics with count 0 pipelines names [].
[22/Jun/2025 13:40:44 +0000] 16316 Metadata-Plugin navigator_plugin INFO Pipelines updated for Metadata Plugin: []
[22/Jun/2025 13:40:47 +0000] 16316 MainThread process INFO [4550-host-inspector] Unregistered supervisor process EXITED
[22/Jun/2025 13:40:49 +0000] 16316 MainThread supervisor INFO Triggering supervisord update.
[22/Jun/2025 13:41:02 +0000] 16316 Thread-16 process INFO [4550-host-inspector] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[22/Jun/2025 13:41:02 +0000] 16316 Thread-16 process INFO [4550-host-inspector] Deactivating process (skipped)
[22/Jun/2025 13:41:02 +0000] 16316 Thread-16 process INFO [4550-host-inspector] stopping monitors
[22/Jun/2025 13:41:03 +0000] 16316 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for host-inspector with count 0 pipelines names [].
[22/Jun/2025 13:41:04 +0000] 16316 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for host-inspector with count 0 pipelines names [].

[#7]

Hello @upadhyayk04 

The following information was found in the /var/log/cloudera-scm-agent/status-stdout.log log file.

   - - [18/May/2025:13:01:02] "GET /process/2693-hdfs-FAILOVERCONTROLLER/files/logs/stderr.log HTTP/1.1" 200 17982 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:02] "GET /process/4475-collect-host-statistics/files/logs/stderr.log HTTP/1.1" 200 49506 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:02] "GET /process/2701-hdfs-HTTPFS/files/logs/stdout.log HTTP/1.1" 206 43782 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:02] "GET /process/4207-hdfs-NAMENODE/files/logs/stdout.log HTTP/1.1" 206 333 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:02] "GET /process/355-zookeeper-server/files/logs/stderr.log HTTP/1.1" 200 20521 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:02] "GET /process/2701-hdfs-HTTPFS/files/logs/stderr.log HTTP/1.1" 200 16034 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:02] "GET /process/4207-hdfs-NAMENODE/files/logs/stderr.log HTTP/1.1" 200 22930 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:02] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1747543200012&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fhadoop-cmf-yarn-NODEMANAGER-  .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/2704-hdfs-DATANODE/files/logs/stdout.log HTTP/1.1" 206 331 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fstacks%2Fhadoop-cmf-hdfs-NAMENODE-  .log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1934 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/4209-yarn-NODEMANAGER/files/logs/stdout.log HTTP/1.1" 206 474 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/4480-host-inspector/files/logs/stdout.log HTTP/1.1" 206 6878 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/2704-hdfs-DATANODE/files/logs/stderr.log HTTP/1.1" 200 22914 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/4209-yarn-NODEMANAGER/files/logs/stderr.log HTTP/1.1" 200 24954 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /download_log?path=%2Fvar%2Flog%2Fzookeeper%2Fstacks%2Fzookeeper-cmf-zookeeper-SERVER-  .log.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1924 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/4480-host-inspector/files/logs/stderr.log HTTP/1.1" 200 7256 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fstacks%2Fhadoop-cmf-yarn-RESOURCEMANAGER-  .log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1948 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.yarn_-7702462360475554876/files/logs/stdout.log HTTP/1.1" 206 561 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_-8215012917422850757/files/logs/stdout.log HTTP/1.1" 206 563 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1684164223045907750/files/logs/stdout.log HTTP/1.1" 206 1209 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.yarn_-7702462360475554876/files/logs/stderr.log HTTP/1.1" 200 20096 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1684164223045907750/files/logs/stderr.log HTTP/1.1" 200 340011 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:05] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_-8215012917422850757/files/logs/stderr.log HTTP/1.1" 200 18892 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:06] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1747543200012&search_timeout_millis=60000&log_path=%2Fvar%2Flog%2Fzookeeper%2Fzookeeper-cmf-zookeeper-SERVER-  .log&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:09] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1747543200012&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-FAILOVERCONTROLLER-  .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:10] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1747543200012&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fhadoop-cmf-yarn-RESOURCEMANAGER-  .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:13] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1747543200012&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-DATANODE-  .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:17] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1747543200012&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-NAMENODE-  .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:17] "GET /compressed_agent_logs?max_bytes=29262586&top_level_dir=  -10.100.165.154 HTTP/1.1" 200 1450090 "" "Java/1.8.0_162"

  - - [18/May/2025:13:01:20] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1747543200012&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-httpfs%2Fhadoop-cmf-hdfs-HTTPFS-  .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [25/May/2025:12:49:20] "GET /process/4489-collect-host-statistics/files/host_statistics/  -13e4bede-bd1e-40e5-b1c9-a47364016427-10.100.165.154-host-statistics.zip HTTP/1.1" 200 829178 "" "Java/1.8.0_162"

  - - [25/May/2025:13:26:58] "GET /process/4494-host-inspector/files/inspector HTTP/1.1" 200 3204 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:40] "GET /process/355-zookeeper-server/files/logs/stderr.log HTTP/1.1" 200 20521 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:40] "GET /process/2701-hdfs-HTTPFS/files/logs/stderr.log HTTP/1.1" 200 16034 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:40] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1748148000011&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fhadoop-cmf-yarn-NODEMANAGER- .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /process/4494-host-inspector/files/logs/stdout.log HTTP/1.1" 206 6879 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /process/4207-hdfs-NAMENODE/files/logs/stdout.log HTTP/1.1" 206 333 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /process/2704-hdfs-DATANODE/files/logs/stdout.log HTTP/1.1" 206 331 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /process/4209-yarn-NODEMANAGER/files/logs/stdout.log HTTP/1.1" 206 474 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /process/4207-hdfs-NAMENODE/files/logs/stderr.log HTTP/1.1" 200 22930 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /process/4494-host-inspector/files/logs/stderr.log HTTP/1.1" 200 7256 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /process/4209-yarn-NODEMANAGER/files/logs/stderr.log HTTP/1.1" 200 24954 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /process/2704-hdfs-DATANODE/files/logs/stderr.log HTTP/1.1" 200 22914 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fstacks%2Fhadoop-cmf-hdfs-NAMENODE- .log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1934 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /download_log?path=%2Fvar%2Flog%2Fzookeeper%2Fstacks%2Fzookeeper-cmf-zookeeper-SERVER- .log.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1924 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:41] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fstacks%2Fhadoop-cmf-yarn-RESOURCEMANAGER- .log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1948 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:42] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.yarn_-7702462360475554876/files/logs/stdout.log HTTP/1.1" 206 561 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:42] "GET /process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1684164223045907750/files/logs/stdout.log HTTP/1.1" 206 1209 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:42] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_-8215012917422850757/files/logs/stdout.log HTTP/1.1" 206 563 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:42] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.yarn_-7702462360475554876/files/logs/stderr.log HTTP/1.1" 200 20096 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:42] "GET /process/ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_-8215012917422850757/files/logs/stderr.log HTTP/1.1" 200 18892 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:42] "GET /process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_1684164223045907750/files/logs/stderr.log HTTP/1.1" 200 340011 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:43] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1748148000011&search_timeout_millis=60000&log_path=%2Fvar%2Flog%2Fzookeeper%2Fzookeeper-cmf-zookeeper-SERVER- .log&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:47] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1748148000011&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-FAILOVERCONTROLLER- .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:48] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1748148000011&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fhadoop-cmf-yarn-RESOURCEMANAGER- .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:51] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1748148000011&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-DATANODE- .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:55] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1748148000011&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-NAMENODE- .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:56] "GET /compressed_agent_logs?max_bytes=29262586&top_level_dir= -10.100.165.154 HTTP/1.1" 200 1450222 "" "Java/1.8.0_162"

  - - [25/May/2025:13:27:58] "GET /retrieve_log_compressed?byte_limit=29262586&end_time=1748148000011&search_timeout_millis=60000&log_path=%2Fdata%2Fvar%2Flog%2Fhadoop-httpfs%2Fhadoop-cmf-hdfs-HTTPFS- .log.out&log_type=LOG4J HTTP/1.1" 200 - "" "Java/1.8.0_162"

  - - [01/Jun/2025:12:49:29] "GET /process/4503-collect-host-statistics/files/host_statistics/ -13e4bede-bd1e-40e5-b1c9-a47364016427-10.100.165.154-host-statistics.zip HTTP/1.1" 200 842280 "" "Java/1.8.0_162"

  - - [01/Jun/2025:13:40:54] "GET /process/4508-host-inspector/files/inspector HTTP/1.1" 200 3203 "" "Java/1.8.0_162"

  - - [01/Jun/2025:13:41:22] "GET /download_log?path=%2Fdata%2Fvar%2Flog%2Fhadoop-yarn%2Fstacks%2Fhadoop-cmf-yarn-NODEMANAGER- .log.out.stacks&onlyTail=false&compress=true HTTP/1.1" 404 1940 "" "Java/1.8.0_162"

[#8]

Hello @Jecky 

Was a reboot performed on the node? Can it be tried once?

Also this might need further deep level analysis regarding the threads so if you have a Cloudera license would it be possible to raise a support case for this

Thank you

Kshitij Upahdyay