Skip to main content
iamfromsky
Gold ⭐️
November 15, 2019
Solved

Cloudera Manager agent bad healthy

my  CDH Cluster has been online more than half year,  all the machines and service is normal.

 

but recently , there is one host agent always under bad healthy, and Cloudera Manager will show the red color on this host.

 

then I check the agent log as below:

[16/Nov/2019 02:20:11 +0000] 1142 MainThread agent WARNING Long HB processing time: 9.52771306038
[16/Nov/2019 02:20:52 +0000] 1142 MainThread agent WARNING Long HB processing time: 34.8044610023
[16/Nov/2019 02:20:52 +0000] 1142 MainThread agent WARNING Delayed HB: 19s since last
[16/Nov/2019 02:21:14 +0000] 1142 MainThread agent WARNING Long HB processing time: 22.3401939869
[16/Nov/2019 02:21:14 +0000] 1142 MainThread agent WARNING Delayed HB: 7s since last
[16/Nov/2019 02:21:51 +0000] 1142 MainThread agent WARNING Long HB processing time: 36.7459609509
[16/Nov/2019 02:21:51 +0000] 1142 MainThread agent WARNING Delayed HB: 21s since last
[16/Nov/2019 02:22:24 +0000] 1142 MainThread agent WARNING Long HB processing time: 33.0430369377
[16/Nov/2019 02:22:24 +0000] 1142 MainThread agent WARNING Delayed HB: 18s since last
[16/Nov/2019 02:23:04 +0000] 1142 MainThread agent WARNING Long HB processing time: 40.0112349987
[16/Nov/2019 02:23:04 +0000] 1142 MainThread agent WARNING Delayed HB: 25s since last
[16/Nov/2019 02:23:35 +0000] 1142 MainThread agent WARNING Long HB processing time: 30.6218628883
[16/Nov/2019 02:23:35 +0000] 1142 MainThread agent WARNING Delayed HB: 15s since last
[16/Nov/2019 02:24:17 +0000] 1142 MainThread agent WARNING Long HB processing time: 42.0491158962
[16/Nov/2019 02:24:17 +0000] 1142 MainThread agent WARNING Delayed HB: 27s since last
[16/Nov/2019 02:24:55 +0000] 1142 MainThread heartbeat_tracker INFO HB stats (seconds): num:17 LIFE_MIN:0.01 min:0.02 mean:0.18 max:0.53 LIFE_MAX:1.67
[16/Nov/2019 02:24:56 +0000] 1142 MainThread agent WARNING Long HB processing time: 38.8551709652
[16/Nov/2019 02:24:56 +0000] 1142 MainThread agent WARNING Delayed HB: 23s since last
[16/Nov/2019 02:25:21 +0000] 1142 MainThread agent WARNING Long HB processing time: 25.7106430531
[16/Nov/2019 02:25:21 +0000] 1142 MainThread agent WARNING Delayed HB: 10s since last
[16/Nov/2019 02:29:03 +0000] 1142 MainThread agent WARNING Long HB processing time: 169.729107141
[16/Nov/2019 02:29:03 +0000] 1142 MainThread agent WARNING Delayed HB: 154s since last
[16/Nov/2019 02:33:56 +0000] 1142 MainThread agent WARNING Long HB processing time: 292.624284029
[16/Nov/2019 02:33:56 +0000] 1142 MainThread agent WARNING Delayed HB: 277s since last

 

I am going to check the server cpu load and other things, it seems nothing happened. then I restart agent, but it will happen hours later or one day later , so my questions is:

1) how this error happened ?

 

thanks

 

 

 

lwang
lwangAnswer
Platinum ⭐️
December 10, 2019

Hi @iamfromsky ,

 

Thanks for keeping us posted about the issue. 

 

Li Wang, Technical Solution Manager

17 replies

lwang
Platinum ⭐️
Platinum ⭐️
November 15, 2019

Hi @iamfromsky ,

 

Could you please share your CDH/CM version?

As you mentioned, I would definitely check the resources (CPU, memory, free spaces, /var/log/messages, network etc) on that host.

Also, check service monitor host and service monitor logs and see if that reveals anything. 

 

Thanks,

Li

Translate
Li Wang, Technical Solution Manager
iamfromsky
Gold ⭐️
December 3, 2019
04/Dec/2019 01:14:54 +0000] 28842 MainThread agent        WARNING  Long HB processing time: 12.050262928
[04/Dec/2019 01:15:03 +0000] 28842 MainThread agent WARNING Long HB processing time: 6.31600308418
[04/Dec/2019 01:15:18 +0000] 28842 MainThread agent WARNING Long HB processing time: 5.97873497009
[04/Dec/2019 01:15:34 +0000] 28842 MainThread agent WARNING Long HB processing time: 7.34343314171
[04/Dec/2019 01:15:54 +0000] 28842 MainThread agent WARNING Long HB processing time: 12.1208558083
[04/Dec/2019 01:16:14 +0000] 28842 MainThread agent WARNING Long HB processing time: 16.9043488503
[04/Dec/2019 01:16:26 +0000] 28842 MainThread agent WARNING Long HB processing time: 12.2732441425
[04/Dec/2019 01:16:47 +0000] 28842 MainThread agent WARNING Long HB processing time: 18.1826350689
[04/Dec/2019 01:16:47 +0000] 28842 MainThread agent WARNING Delayed HB: 3s since last
[04/Dec/2019 01:16:49 +0000] 28842 MonitorDaemon-Reporter throttling_logger WARNING Failed to get swap memory usage for process 13345: list index out of range
[04/Dec/2019 01:18:02 +0000] 28842 MainThread agent WARNING Long HB processing time: 15.3277680874
[04/Dec/2019 01:18:27 +0000] 28842 MainThread agent WARNING Long HB processing time: 9.59771895409
[04/Dec/2019 01:18:47 +0000] 28842 MainThread agent WARNING Long HB processing time: 14.4301588535
[04/Dec/2019 01:18:53 +0000] 28842 MainThread agent WARNING Long HB processing time: 5.77177095413
[04/Dec/2019 01:19:17 +0000] 28842 MainThread agent WARNING Long HB processing time: 14.8392968178
[04/Dec/2019 01:19:33 +0000] 28842 MainThread agent WARNING Long HB processing time: 15.5372338295
[04/Dec/2019 01:20:50 +0000] 28842 MainThread agent WARNING Long HB processing time: 46.7583889961
[04/Dec/2019 01:20:50 +0000] 28842 MainThread agent WARNING Delayed HB: 31s since last
[04/Dec/2019 01:22:24 +0000] 28842 MainThread heartbeat_tracker INFO HB stats (seconds): num:31 LIFE_MIN:0.00 min:0.02 mean:0.28 max:2.14 LIFE_MAX:4.53
[04/Dec/2019 01:22:25 +0000] 28842 MainThread throttling_logger INFO (12 skipped) Identified java component java8 with full version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) for requested version 8.
[04/Dec/2019 01:22:26 +0000] 28842 MainThread agent WARNING Long HB processing time: 95.7612428665
[04/Dec/2019 01:22:26 +0000] 28842 MainThread agent WARNING Delayed HB: 80s since last
[04/Dec/2019 01:22:32 +0000] 28842 MainThread agent WARNING Long HB processing time: 6.20306706429
[04/Dec/2019 01:24:18 +0000] 28842 MainThread agent WARNING Long HB processing time: 97.3476579189
[04/Dec/2019 01:24:18 +0000] 28842 MainThread agent WARNING Delayed HB: 82s since last
[04/Dec/2019 01:25:32 +0000] 28842 MainThread agent WARNING Long HB processing time: 73.8358900547
[04/Dec/2019 01:25:32 +0000] 28842 MainThread agent WARNING Delayed HB: 58s since last
[04/Dec/2019 01:26:21 +0000] 28842 MainThread agent WARNING Long HB processing time: 48.3692760468
[04/Dec/2019 01:26:21 +0000] 28842 MainThread agent WARNING Delayed HB: 33s since last
[04/Dec/2019 01:28:50 +0000] 28842 MainThread agent WARNING Long HB processing time: 149.57425499
[04/Dec/2019 01:28:50 +0000] 28842 MainThread agent WARNING Delayed HB: 134s since last
[04/Dec/2019 01:30:42 +0000] 28842 MainThread agent WARNING Long HB processing time: 111.463696957
[04/Dec/2019 01:30:42 +0000] 28842 MainThread agent WARNING Delayed HB: 96s since last
[04/Dec/2019 01:30:47 +0000] 28842 MainThread agent WARNING Long HB processing time: 5.08097100258
[04/Dec/2019 01:32:58 +0000] 28842 MainThread heartbeat_tracker INFO HB stats (seconds): num:8 LIFE_MIN:0.00 min:0.11 mean:0.71 max:1.96 LIFE_MAX:4.53
[04/Dec/2019 01:33:00 +0000] 28842 MainThread agent WARNING Long HB processing time: 122.480878115
[04/Dec/2019 01:33:00 +0000] 28842 MainThread agent WARNING Delayed HB: 107s since last
[04/Dec/2019 01:34:10 +0000] 28842 MainThread agent WARNING Long HB processing time: 69.9775719643
[04/Dec/2019 01:34:10 +0000] 28842 MainThread agent WARNING Delayed HB: 55s since last
[04/Dec/2019 01:35:45 +0000] 28842 MainThread agent WARNING Long HB processing time: 95.0239050388
[04/Dec/2019 01:35:45 +0000] 28842 MainThread agent WARNING Delayed HB: 80s since last
[04/Dec/2019 01:37:42 +0000] 28842 MainThread agent WARNING Long HB processing time: 117.279044151
[04/Dec/2019 01:37:42 +0000] 28842 MainThread agent WARNING Delayed HB: 102s since last
[04/Dec/2019 01:40:33 +0000] 28842 MainThread agent WARNING Long HB processing time: 170.700104952
[04/Dec/2019 01:40:33 +0000] 28842 MainThread agent WARNING Delayed HB: 155s since last
[04/Dec/2019 01:40:43 +0000] 28842 MainThread agent WARNING Long HB processing time: 10.2622208595
[04/Dec/2019 01:41:17 +0000] 28842 MainThread agent WARNING Long HB processing time: 28.4504780769
[04/Dec/2019 01:41:17 +0000] 28842 MainThread agent WARNING Delayed HB: 13s since last
[04/Dec/2019 01:43:17 +0000] 28842 MainThread heartbeat_tracker INFO HB stats (seconds): num:7 LIFE_MIN:0.00 min:0.02 mean:0.94 max:1.48 LIFE_MAX:4.53
[04/Dec/2019 01:43:18 +0000] 28842 MainThread agent WARNING Long HB processing time: 121.649900913
[04/Dec/2019 01:43:18 +0000] 28842 MainThread agent WARNING Delayed HB: 106s since last
[04/Dec/2019 01:44:11 +0000] 28842 MainThread agent WARNING Long HB processing time: 52.2356939316
[04/Dec/2019 01:44:11 +0000] 28842 MainThread agent WARNING Delayed HB: 37s since last
[04/Dec/2019 01:44:28 +0000] 28842 MainThread agent WARNING Long HB processing time: 17.3855171204
[04/Dec/2019 01:47:23 +0000] 28842 MainThread agent WARNING Long HB processing time: 175.126209021
[04/Dec/2019 01:47:24 +0000] 28842 MainThread agent WARNING Delayed HB: 160s since last
[04/Dec/2019 01:48:27 +0000] 28842 MainThread agent WARNING Long HB processing time: 63.2303080559
[04/Dec/2019 01:48:27 +0000] 28842 MainThread agent WARNING Delayed HB: 48s since last
[04/Dec/2019 01:49:00 +0000] 28842 MonitorDaemon-Reporter throttling_logger ERROR (22 skipped) Role_extractor for STATESTORE is not initialized correctly.
None
[04/Dec/2019 01:49:12 +0000] 28842 MainThread agent WARNING Long HB processing time: 45.4053838253
[04/Dec/2019 01:49:13 +0000] 28842 MainThread agent WARNING Delayed HB: 31s since last
[04/Dec/2019 01:51:19 +0000] 28842 MainThread agent WARNING Long HB processing time: 125.514182091
[04/Dec/2019 01:51:19 +0000] 28842 MainThread agent WARNING Delayed HB: 110s since last
[04/Dec/2019 01:52:38 +0000] 28842 MainThread agent WARNING Long HB processing time: 78.3114128113
[04/Dec/2019 01:52:38 +0000] 28842 MainThread agent WARNING Delayed HB: 63s since last
[04/Dec/2019 01:52:45 +0000] 28842 MainThread agent WARNING Long HB processing time: 6.34388899803
[04/Dec/2019 01:53:03 +0000] 28842 MainThread agent WARNING Long HB processing time: 9.70307207108
[04/Dec/2019 01:53:30 +0000] 28842 MainThread heartbeat_tracker INFO HB stats (seconds): num:10 LIFE_MIN:0.00 min:0.06 mean:0.58 max:1.79 LIFE_MAX:4.53
[04/Dec/2019 01:53:30 +0000] 28842 MainThread throttling_logger INFO (8 skipped) Identified java component java8 with full version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) for requested version 8.
[04/Dec/2019 01:53:30 +0000] 28842 MainThread agent WARNING Long HB processing time: 21.8870470524
[04/Dec/2019 01:53:30 +0000] 28842 MainThread agent WARNING Delayed HB: 6s since last
[04/Dec/2019 01:53:46 +0000] 28842 MainThread agent WARNING Long HB processing time: 16.0326740742
[04/Dec/2019 01:54:08 +0000] 28842 MainThread agent WARNING Long HB processing time: 22.1611609459
[04/Dec/2019 01:54:08 +0000] 28842 MainThread agent WARNING Delayed HB: 7s since last
[04/Dec/2019 01:54:14 +0000] 28842 MainThread agent WARNING Long HB processing time: 6.01884293556
[04/Dec/2019 01:54:34 +0000] 28842 MainThread agent WARNING Long HB processing time: 10.3302299976
[04/Dec/2019 01:54:45 +0000] 28842 MainThread agent WARNING Long HB processing time: 6.26238012314
[04/Dec/2019 01:54:59 +0000] 28842 MainThread agent WARNING Long HB processing time: 5.61235713959
[04/Dec/2019 01:55:48 +0000] 28842 MainThread agent WARNING Long HB processing time: 39.5674409866
[04/Dec/2019 01:55:48 +0000] 28842 MainThread agent WARNING Delayed HB: 24s since last
[04/Dec/2019 01:56:04 +0000] 28842 MainThread agent WARNING Long HB processing time: 15.7047331333
[04/Dec/2019 01:56:45 +0000] 28842 MainThread agent WARNING Long HB processing time: 26.3552570343
[04/Dec/2019 01:56:45 +0000] 28842 MainThread agent WARNING Delayed HB: 11s since last
[04/Dec/2019 01:58:15 +0000] 28842 MainThread agent WARNING Long HB processing time: 44.8799829483
[04/Dec/2019 01:58:15 +0000] 28842 MainThread agent WARNING Delayed HB: 29s since last
[04/Dec/2019 01:59:02 +0000] 28842 MainThread agent WARNING Long HB processing time: 47.1872057915
[04/Dec/2019 01:59:02 +0000] 28842 MainThread agent WARNING Delayed HB: 32s since last
[04/Dec/2019 01:59:17 +0000] 28842 MainThread agent WARNING Long HB processing time: 14.3799068928
[04/Dec/2019 02:00:27 +0000] 28842 MainThread agent WARNING Long HB processing time: 54.5242679119
[04/Dec/2019 02:00:27 +0000] 28842 MainThread agent WARNING Delayed HB: 39s since last
[04/Dec/2019 02:01:07 +0000] 28842 MonitorDaemon-Reporter throttling_logger INFO (10 skipped) Descendants user CPU lower than expected for process 7167: 42154365.0, 42134965.36
[04/Dec/2019 02:02:45 +0000] 28842 MainThread agent WARNING Long HB processing time: 137.669925213
[04/Dec/2019 02:02:45 +0000] 28842 MainThread agent WARNING Delayed HB: 122s since last
[04/Dec/2019 02:04:41 +0000] 28842 MainThread heartbeat_tracker INFO HB stats (seconds): num:20 LIFE_MIN:0.00 min:0.02 mean:0.52 max:2.48 LIFE_MAX:4.53
[04/Dec/2019 02:04:42 +0000] 28842 MainThread agent WARNING Long HB processing time: 117.50967598
[04/Dec/2019 02:04:42 +0000] 28842 MainThread agent WARNING Delayed HB: 102s since last
[04/Dec/2019 02:07:01 +0000] 28842 MainThread agent WARNING Long HB processing time: 138.424443007
[04/Dec/2019 02:07:01 +0000] 28842 MainThread agent WARNING Delayed HB: 123s since last
[04/Dec/2019 02:09:43 +0000] 28842 MainThread agent WARNING Long HB processing time: 162.216438055
[04/Dec/2019 02:09:44 +0000] 28842 MainThread agent WARNING Delayed HB: 147s since last
[04/Dec/2019 02:11:34 +0000] 28842 MainThread agent WARNING Long HB processing time: 109.956519842
[04/Dec/2019 02:11:34 +0000] 28842 MainThread agent WARNING Delayed HB: 95s since last
[04/Dec/2019 02:12:10 +0000] 28842 MainThread agent WARNING Long HB processing time: 36.4911990166
[04/Dec/2019 02:12:11 +0000] 28842 MainThread agent WARNING Delayed HB: 21s since last
[04/Dec/2019 02:12:17 +0000] 28842 MainThread agent WARNING Long HB processing time: 6.5263209343
[04/Dec/2019 02:13:37 +0000] 28842 MainThread agent WARNING Long HB processing time: 71.4897689819
[04/Dec/2019 02:13:37 +0000] 28842 MainThread agent WARNING Delayed HB: 56s since last
[04/Dec/2019 02:15:11 +0000] 28842 MainThread heartbeat_tracker INFO HB stats (seconds): num:7 LIFE_MIN:0.00 min:0.20 mean:1.26 max:1.96 LIFE_MAX:4.53
[04/Dec/2019 02:15:13 +0000] 28842 MainThread agent WARNING Long HB processing time: 95.5481140614
[04/Dec/2019 02:15:13 +0000] 28842 MainThread agent WARNING Delayed HB: 80s since last
[04/Dec/2019 02:17:41 +0000] 28842 MainThread agent WARNING Long HB processing time: 147.541055918
[04/Dec/2019 02:17:41 +0000] 28842 MainThread agent WARNING Delayed HB: 132s since last
[04/Dec/2019 02:17:52 +0000] 28842 MainThread agent WARNING Long HB processing time: 11.1138081551
[04/Dec/2019 02:18:09 +0000] 28842 MainThread agent WARNING Long HB processing time: 13.3887140751
[04/Dec/2019 02:21:01 +0000] 28842 MainThread agent WARNING Long HB processing time: 170.292068958
[04/Dec/2019 02:21:01 +0000] 28842 MainThread agent WARNING Delayed HB: 155s since last
[04/Dec/2019 02:21:11 +0000] 28842 MainThread agent WARNING Long HB processing time: 9.28128290176
[04/Dec/2019 02:26:27 +0000] 28842 MainThread agent ERROR Heartbeating to 10.203.3.97:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1396, in _send_heartbeat
response = self.requestor.request('heartbeat', heartbeat_data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
result = self.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
response = self.conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
[04/Dec/2019 02:26:27 +0000] 28842 MainThread heartbeat_tracker INFO HB stats (seconds): num:6 LIFE_MIN:0.00 min:0.02 mean:0.62 max:1.55 LIFE_MAX:4.53
[04/Dec/2019 02:26:27 +0000] 28842 MainThread agent WARNING Long HB processing time: 310.67294383
[04/Dec/2019 02:26:28 +0000] 28842 MainThread agent WARNING Delayed HB: 295s since last
[04/Dec/2019 02:30:28 +0000] 28842 MainThread throttling_logger INFO (10 skipped) Identified java component java8 with full version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) for requested version 8.
[04/Dec/2019 02:30:29 +0000] 28842 MainThread agent WARNING Long HB processing time: 241.416838169
[04/Dec/2019 02:30:29 +0000] 28842 MainThread agent WARNING Delayed HB: 226s since last
[04/Dec/2019 02:37:11 +0000] 28842 MainThread agent ERROR Heartbeating to 10.203.3.97:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1396, in _send_heartbeat
response = self.requestor.request('heartbeat', heartbeat_data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
result = self.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
response = self.conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
[04/Dec/2019 02:37:11 +0000] 28842 MainThread heartbeat_tracker INFO HB stats (seconds): num:2 LIFE_MIN:0.00 min:0.44 mean:0.48 max:0.53 LIFE_MAX:4.53
[04/Dec/2019 02:37:11 +0000] 28842 MainThread agent WARNING Long HB processing time: 402.084249973
[04/Dec/2019 02:37:11 +0000] 28842 MainThread agent WARNING Delayed HB: 387s since last

 

could you get useful information from the above logs ?  

 

I can see the logs show me heartbeat failed, but I don't know why ? could you give me some advises? 

 

thank you very much .

Translate
zuoseven
Silver ⭐️⭐️
August 3, 2023

jus restart impala role and fix this,bucause there have manny tcp conntion

WX20230803-174634.png

Translate