question Re: Getting Error while running the ansible play book for deploying CDP in Support Questions

Getting Error while running the ansible play book for deploying CDP

Robin1986 — Tue, 21 Apr 2026 06:43:55 GMT

I am new to the Cloudera CDP and got stuck at the CMS level

Re: Getting Error while running the ansible play book for deploying CDP

james_jones — Wed, 17 Jan 2024 14:41:37 GMT

Check for errors the Cloudera Manager server log file in /var/log/cloudera-scm-server/cloudera-scm-server.log.

Also, I see that the URL is nitbucc-vad001:7180. You should use fully qualified domain names (e.g. nitbucc-vad001.xyz.local) for hosts and in the playbook configurations. I'm not saying that is the issue, but you may run into issues later if you don't use FQDN (especially if you secure the cluster).

Re: Getting Error while running the ansible play book for deploying CDP

Robin1986 — Thu, 18 Jan 2024 12:06:28 GMT

cloudera-scm-agent Error log

[18/Jan/2024 11:13:12 +0100] 28630 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[18/Jan/2024 11:13:14 +0100] 28630 MainThread supervisor INFO Trying to connect to supervisor (Attempt 1)
[18/Jan/2024 11:13:14 +0100] 28630 MainThread supervisor INFO Supervisor version: 3.4.0, pid: 28629
[18/Jan/2024 11:13:14 +0100] 28630 MainThread supervisor INFO Successfully connected to supervisor
[18/Jan/2024 11:13:14 +0100] 28630 MainThread agent INFO Supervisor version: 3.4.0, pid: 28629
[18/Jan/2024 11:13:14 +0100] 28630 MainThread agent INFO Connecting to previous supervisor: agent-28629-1705572792.
[18/Jan/2024 11:13:14 +0100] 28630 MainThread supervisor INFO Triggering supervisord update.
[18/Jan/2024 11:13:14 +0100] 28630 MainThread _cplogging INFO [18/Jan/2024:11:13:14] ENGINE Bus STARTING
[18/Jan/2024 11:13:14 +0100] 28630 MainThread _cplogging INFO [18/Jan/2024:11:13:14] ENGINE Started monitor thread '_TimeoutMonitor'.
[18/Jan/2024 11:13:14 +0100] 28630 MainThread _cplogging INFO [18/Jan/2024:11:13:14] ENGINE Serving on http://127.0.0.1:9001
[18/Jan/2024 11:13:14 +0100] 28630 MainThread _cplogging INFO [18/Jan/2024:11:13:14] ENGINE Bus STARTED
[18/Jan/2024 11:13:14 +0100] 28630 MainThread status_server INFO Status server url is http://nitucc-vad001.data.nitucc:9000/
[18/Jan/2024 11:13:14 +0100] 28630 MainThread daemon INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x7f77f7884d90>,)
[18/Jan/2024 11:13:14 +0100] 28630 MonitorDaemon-Scheduler daemon INFO Monitor ready to report: ('HostMonitor',)
[18/Jan/2024 11:13:14 +0100] 28630 MainThread agent INFO Setting default socket timeout to 45
[18/Jan/2024 11:13:14 +0100] 28630 MainThread agent INFO Failed to read available parcel file: [Errno 2] No such file or directory: '/var/lib/cloudera-scm-agent/active_parcels.json'
[18/Jan/2024 11:13:14 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:14 +0100] 28630 MainThread agent ERROR Heartbeating to nitucc-vad001.data.nitucc:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1453, in _send_heartbeat
self.cfg.master_port)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__
self.conn.connect()
File "/usr/lib/python2.7/httplib.py", line 837, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 111] Connection refused
[18/Jan/2024 11:13:14 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.00 min:0.00 mean:0.00 max:0.00 LIFE_MAX:0.00
[18/Jan/2024 11:13:14 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:17 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:17 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:22 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:27 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:32 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:37 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:42 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:47 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:52 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:13:57 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:02 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:07 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:12 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:17 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:17 +0100] 28630 MainThread agent ERROR Heartbeating to nitucc-vad001.data.nitucc:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1453, in _send_heartbeat
self.cfg.master_port)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__
self.conn.connect()
File "/usr/lib/python2.7/httplib.py", line 837, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 111] Connection refused
[18/Jan/2024 11:14:22 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:27 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:32 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:37 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:42 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:47 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:52 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:14:57 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:15:02 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO CM server guid: 935ce6b8-11ca-4e04-ba34-38f66ace60f9
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO Created /opt/cloudera/parcels
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO Chowning /opt/cloudera/parcels to root (0) root (0)
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO Chmod'ing /opt/cloudera/parcels to 0755
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO Created /opt/cloudera/parcel-cache
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO Chowning /opt/cloudera/parcel-cache to root (0) root (0)
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO Chmod'ing /opt/cloudera/parcel-cache to 0755
[18/Jan/2024 11:15:07 +0100] 28630 MainThread parcel INFO Agent does create users/groups
[18/Jan/2024 11:15:07 +0100] 28630 MainThread parcel INFO Agent does parcel permissions
[18/Jan/2024 11:15:07 +0100] 28630 MainThread agent INFO Allocating static UIDs/GIDs is enabled.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Begin to init GID/UID data structure in agents
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Reading old groups.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Finished reading 82 old groups.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Reading old services.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Finished reading 57 old services.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Matching old groups to services.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Finished matching 15 old groups to 18 services.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Reading new groups from file /opt/cloudera/cm-agent/service/inituids/group_list.csv.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Finished reading 1 new groups. Matched 0 old groups.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Reading new services from file /opt/cloudera/cm-agent/service/inituids/service_list.csv.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Finished reading 40 new services. Matched 0 old services.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Apply service ids after initialization
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Checking if services already exist.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids WARNING Found 17 non-matching, and 0 matching, existing services. Do you want to re-run with migration flag? Hit CTRL-C to stop now.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids WARNING Going ahead with initialization of any missing services. Existing services will be ignored.
[18/Jan/2024 11:15:07 +0100] 28630 MainThread setServiceUids INFO Creating 23 new groups.
[18/Jan/2024 11:15:11 +0100] 28630 MainThread setServiceUids INFO Finished creating groups: Created 22, Existing 1, Failed 0.
[18/Jan/2024 11:15:11 +0100] 28630 MainThread setServiceUids INFO Creating 40 new service accounts.
[18/Jan/2024 11:15:11 +0100] 28630 MainThread setServiceUids INFO Adding existing user hdfs to groups: hadoop
[18/Jan/2024 11:15:11 +0100] 28630 MainThread setServiceUids INFO Adding existing user yarn to groups: hadoop
[18/Jan/2024 11:15:13 +0100] 28630 MainThread setServiceUids INFO Adding existing user mapred to groups: hadoop
[18/Jan/2024 11:15:14 +0100] 28630 MainThread setServiceUids INFO Adding existing user impala to groups: hive
[18/Jan/2024 11:15:16 +0100] 28630 MainThread setServiceUids INFO Finished creating services: Created 23, Existing 17, Failed 0.
[18/Jan/2024 11:15:16 +0100] 28630 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[18/Jan/2024 11:15:16 +0100] 28630 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[18/Jan/2024 11:15:16 +0100] 28630 MainThread throttling_logger WARNING Failed parsing alternatives line: pager string index out of range link best version is /usr/bin/less
[18/Jan/2024 11:15:17 +0100] 28630 MainThread agent INFO Flood daemon (re)start attempt
[18/Jan/2024 11:15:17 +0100] 28630 MainThread agent INFO Created /opt/cloudera/parcels/.flood
[18/Jan/2024 11:15:17 +0100] 28630 MainThread agent INFO Chowning /opt/cloudera/parcels/.flood to cloudera-scm (117) cloudera-scm (120)
[18/Jan/2024 11:15:17 +0100] 28630 MainThread agent INFO Chmod'ing /opt/cloudera/parcels/.flood to 0755
[18/Jan/2024 11:15:18 +0100] 28630 MainThread agent INFO Reloading status server config, sending SIGHUP
[18/Jan/2024 11:15:18 +0100] 28630 MainThread agent INFO Active parcel list updated; recalculating component info.
[18/Jan/2024 11:15:18 +0100] 28630 MainThread throttling_logger INFO Identified java component java11 with full version java version "11.0.11" 2021-04-20 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.11+9-LTS-194) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.11+9-LTS-194, mixed mode) for requested version .
[18/Jan/2024 11:15:18 +0100] 28630 MainThread agent WARNING Long HB processing time: 11.1835989952
[18/Jan/2024 11:15:19 +0100] 28630 Monitor-HostMonitor network_interfaces WARNING Interface 'eth0' reported unknown speed '4294967295'
[18/Jan/2024 11:15:19 +0100] 28630 Monitor-HostMonitor network_interfaces WARNING Interface 'eth0' reported unknown duplex mode '255'
[18/Jan/2024 11:15:19 +0100] 28630 Monitor-HostMonitor throttling_logger ERROR Failed to collect NTP metrics
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 49, in collect
self.collect_ntpd()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 67, in collect_ntpd
result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 39, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/subprocess_timeout.py", line 50, in subprocess_with_timeout
p = subprocess.Popen(**kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
[18/Jan/2024 11:23:18 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:58 LIFE_MIN:0.00 min:0.00 mean:0.01 max:0.05 LIFE_MAX:0.01
[18/Jan/2024 11:33:19 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.01
[18/Jan/2024 11:42:34 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:42:34 +0100] 28630 MainThread agent ERROR Heartbeating to nitucc-vad001.data.nitucc:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1459, in _send_heartbeat
response = self.requestor.request('heartbeat', heartbeat_data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
result = self.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
response = self.conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1178, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 452, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 416, in _read_status
raise BadStatusLine("No status line received - the server has closed the connection")
BadStatusLine: No status line received - the server has closed the connection
[18/Jan/2024 11:42:49 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:42:49 +0100] 28630 MainThread agent ERROR Heartbeating to nitucc-vad001.data.nitucc:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1453, in _send_heartbeat
self.cfg.master_port)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__
self.conn.connect()
File "/usr/lib/python2.7/httplib.py", line 837, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 111] Connection refused
[18/Jan/2024 11:43:04 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:43:19 +0100] 28630 MainThread agent WARNING Port is being used by CM Agent: 7182
[18/Jan/2024 11:43:19 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.00 mean:0.01 max:0.02 LIFE_MAX:0.01
[18/Jan/2024 11:45:19 +0100] 28630 Monitor-HostMonitor throttling_logger ERROR (359 skipped) Failed to collect NTP metrics
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 49, in collect
self.collect_ntpd()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 67, in collect_ntpd
result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 39, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/subprocess_timeout.py", line 50, in subprocess_with_timeout
p = subprocess.Popen(**kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
[18/Jan/2024 11:45:35 +0100] 28630 MainThread throttling_logger INFO (14 skipped) Identified java component java11 with full version java version "11.0.11" 2021-04-20 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.11+9-LTS-194) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.11+9-LTS-194, mixed mode) for requested version .
[18/Jan/2024 11:53:20 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.01 mean:0.01 max:0.06 LIFE_MAX:0.01
[18/Jan/2024 12:03:20 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.01
[18/Jan/2024 12:13:21 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.01
[18/Jan/2024 12:15:19 +0100] 28630 Monitor-HostMonitor throttling_logger ERROR (359 skipped) Failed to collect NTP metrics
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 49, in collect
self.collect_ntpd()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 67, in collect_ntpd
result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 39, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/subprocess_timeout.py", line 50, in subprocess_with_timeout
p = subprocess.Popen(**kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
[18/Jan/2024 12:15:36 +0100] 28630 MainThread throttling_logger INFO (14 skipped) Identified java component java11 with full version java version "11.0.11" 2021-04-20 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.11+9-LTS-194) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.11+9-LTS-194, mixed mode) for requested version .
[18/Jan/2024 12:23:21 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.01
[18/Jan/2024 12:33:22 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.01
[18/Jan/2024 12:43:22 +0100] 28630 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.01
[18/Jan/2024 12:45:20 +0100] 28630 Monitor-HostMonitor throttling_logger ERROR (359 skipped) Failed to collect NTP metrics
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 49, in collect
self.collect_ntpd()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 67, in collect_ntpd
result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/ntp_monitor.py", line 39, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/subprocess_timeout.py", line 50, in subprocess_with_timeout
p = subprocess.Popen(**kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
[18/Jan/2024 12:45:37 +0100] 28630 MainThread throttling_logger INFO (14 skipped) Identified java component java11 with full version java version "11.0.11" 2021-04-20 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.11+9-LTS-194) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.11+9-LTS-194, mixed mode) for requested version .

Cloudera-scm-server log.

. finalstate:FINISHED, success:true, msg:Successfully finished checking for configuration staleness.
2024-01-18 11:16:10,577 INFO CommandPusher-1:com.cloudera.cmf.command.components.CommandStorage: Invoked delete temp files for command:DbCommand{id=45, name=ProcessStalenessCheckCommand} at dir:/var/lib/cloudera-scm-server/temp/commands/45
2024-01-18 11:16:16,522 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (37 skipped) AgentAvroServlet: heartbeat processing stats: average=19ms, min=2ms, max=119ms.
2024-01-18 11:16:21,725 INFO scm-web-92:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 10.102.11.99
2024-01-18 11:16:52,156 INFO scm-web-95:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 10.102.11.99
2024-01-18 11:16:52,165 INFO scm-web-95:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:PUT, Path:/v51/cm/service
2024-01-18 11:16:52,247 INFO scm-web-95:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:Cannot construct instance of `com.cloudera.api.model.ApiService` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{
"name": "mgmt",
"type": "MGMT",
"roles": [
{
"name": "mgmt-SERVICEMONITOR",
"type": "SERVICEMONITOR",
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}, {
"name": "mgmt-HOSTMONITOR",
"type": "HOSTMONITOR",
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}, {
"name": "mgmt-REPORTSMANAGER",
"type": "REPORTSMANAGER",
"roleConfigGroupRef": {
"roleConfigGroupName": "mgmt-REPORTSMANAGER-BASE"
},
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}, {
"name": "mgmt-EVENTSERVER",
"type": "EVENTSERVER",
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}, {
"name": "mgmt-ALERTPUBLISHER",
"type": "ALERTPUBLISHER",
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}
],
"roleConfigGroups": [
{
"name": "mgmt-REPORTSMANAGER-BASE",
"displayName": "Reports Manager Default Group",
"roleType": "REPORTSMANAGER",
"base": true,
"serviceRef": {
"serviceName": "mgmt"
},
"config": {
"items": [
{
"name": "headlamp_database_host",
"value": "nitucc-vhm001.data.nitucc"
}, {
"name": "headlamp_database_name",
"value": "rman"
}, {
"name": "headlamp_database_password",
"value": "ambariowner"
}, {
"name": "headlamp_database_type",
"value": "mysql"
}, {
"name": "headlamp_database_user",
"value": "rman"
}
]
}
}
]
}')
at [Source: (org.apache.cxf.transport.http.AbstractHTTPDestination$1); line: 1, column: 1]
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `com.cloudera.api.model.ApiService` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('{
"name": "mgmt",
"type": "MGMT",
"roles": [
{
"name": "mgmt-SERVICEMONITOR",
"type": "SERVICEMONITOR",
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}, {
"name": "mgmt-HOSTMONITOR",
"type": "HOSTMONITOR",
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}, {
"name": "mgmt-REPORTSMANAGER",
"type": "REPORTSMANAGER",
"roleConfigGroupRef": {
"roleConfigGroupName": "mgmt-REPORTSMANAGER-BASE"
},
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}, {
"name": "mgmt-EVENTSERVER",
"type": "EVENTSERVER",
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}, {
"name": "mgmt-ALERTPUBLISHER",
"type": "ALERTPUBLISHER",
"hostRef": {
"hostId": "3e533e4a-dc38-457c-92c3-08137c84dafa"
}
}
],
"roleConfigGroups": [
{
"name": "mgmt-REPORTSMANAGER-BASE",
"displayName": "Reports Manager Default Group",
"roleType": "REPORTSMANAGER",
"base": true,
"serviceRef": {
"serviceName": "mgmt"
},
"config": {
"items": [
{
"name": "headlamp_database_host",
"value": "nitucc-vhm001.data.nitucc"
}, {
"name": "headlamp_database_name",
"value": "rman"
}, {
"name": "headlamp_database_password",
"value": "ambariowner"
}, {
"name": "headlamp_database_type",
"value": "mysql"
}, {
"name": "headlamp_database_user",
"value": "rman"
}
]
}
}
]
}')
at [Source: (org.apache.cxf.transport.http.AbstractHTTPDestination$1); line: 1, column: 1]
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1728)
at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1353)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromString(StdDeserializer.java:311)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromString(BeanDeserializerBase.java:1495)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:197)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:2025)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1175)
at com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:803)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1500)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:1452)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.processRequestBodyParameter(JAXRSUtils.java:950)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:881)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:283)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:286)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:286)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPut(AbstractHTTPServlet.java:234)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:204)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.cloudera.server.cmf.config.components.CmfHttpSessionFilter.doFilter(CmfHttpSessionFilter.java:35)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.cloudera.server.cmf.config.components.RequestRecastFilter.doFilter(RequestRecastFilter.java:55)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.kerberos.web.authentication.SpnegoAuthenticationProcessingFilter.doFilter(SpnegoAuthenticationProcessingFilter.java:128)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:86)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:716)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.base/java.lang.Thread.run(Thread.java:834)
2024-01-18 11:16:52,252 INFO scm-web-95:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:PUT, Path:/v51/cm/service, Status:400
2024-01-18 11:16:54,728 INFO scm-web-92:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 10.102.11.99
2024-01-18 11:16:54,731 INFO scm-web-92:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/v51/cm/service/commands/start
2024-01-18 11:16:54,772 INFO scm-web-92:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:Cannot find management service.
java.util.NoSuchElementException: Cannot find management service.
at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getMgmtService(ServiceManagerDaoImpl.java:832)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:264)
at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:95)
at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:300)
at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:131)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:181)
at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:333)
at com.sun.proxy.$Proxy245.getMgmtService(Unknown Source)
at com.cloudera.api.v1.impl.MgmtServiceResourceImpl.readService(MgmtServiceResourceImpl.java:46)
at com.cloudera.api.v1.impl.MgmtServiceResourceImpl.startCommand(MgmtServiceResourceImpl.java:81)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:286)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:286)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:286)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:204)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.cloudera.server.cmf.config.components.CmfHttpSessionFilter.doFilter(CmfHttpSessionFilter.java:35)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.cloudera.server.cmf.config.components.RequestRecastFilter.doFilter(RequestRecastFilter.java:55)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.kerberos.web.authentication.SpnegoAuthenticationProcessingFilter.doFilter(SpnegoAuthenticationProcessingFilter.java:128)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:86)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:716)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.base/java.lang.Thread.run(Thread.java:834)
2024-01-18 11:16:54,776 INFO scm-web-92:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/v51/cm/service/commands/start, Status:404
2024-01-18 11:17:06,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:17:07,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:17:16,580 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=14ms, min=2ms, max=119ms.
2024-01-18 11:18:08,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:18:09,596 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:18:16,638 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=11ms, min=2ms, max=119ms.
2024-01-18 11:19:07,023 INFO ReporterRegistry-0:com.cloudera.cmf.service.scm.components.ReporterRegistry: Triggering recalculation because of event: scm-cpu-usage-changes
2024-01-18 11:19:10,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:19:11,592 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:19:16,697 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=10ms, min=2ms, max=119ms.
2024-01-18 11:20:12,596 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:20:13,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:20:16,757 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=2ms, max=119ms.
2024-01-18 11:21:13,592 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Synced up
2024-01-18 11:21:14,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:21:16,819 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=9ms, min=2ms, max=119ms.
2024-01-18 11:22:15,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:22:16,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:22:16,878 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=8ms, min=2ms, max=119ms.
2024-01-18 11:23:16,931 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=8ms, min=2ms, max=119ms.
2024-01-18 11:23:17,589 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:23:18,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:24:16,990 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=8ms, min=2ms, max=119ms.
2024-01-18 11:24:17,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Synced up
2024-01-18 11:24:20,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:25:05,653 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2024-01-18 11:25:05,660 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2022-01-18T10:25:05.654Z to reap.
2024-01-18 11:25:05,662 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2024-01-18 11:25:17,049 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=8ms, min=2ms, max=119ms.
2024-01-18 11:25:17,050 INFO agentServer-63:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=9ms, min=0ms, max=150ms.
2024-01-18 11:25:17,050 INFO agentServer-63:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=4ms.
2024-01-18 11:25:19,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:25:22,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:26:17,109 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:26:21,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:26:24,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:27:17,168 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:27:23,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:27:24,596 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Cleaned up
2024-01-18 11:28:17,242 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:28:25,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:28:26,594 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:29:17,283 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:29:26,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Cleaned up
2024-01-18 11:29:27,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:30:17,343 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:30:28,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:30:29,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:31:17,399 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:31:30,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:31:31,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:32:17,460 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:32:32,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:32:33,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:33:17,507 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:33:34,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:33:35,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:34:17,566 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:34:36,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:34:37,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:35:05,675 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2024-01-18 11:35:05,677 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2022-01-18T10:35:05.676Z to reap.
2024-01-18 11:35:05,678 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2024-01-18 11:35:17,613 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=7ms, min=2ms, max=119ms.
2024-01-18 11:35:17,613 INFO agentServer-64:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=8ms, min=0ms, max=150ms.
2024-01-18 11:35:17,613 INFO agentServer-64:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=4ms.
2024-01-18 11:35:37,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Synced up
2024-01-18 11:35:38,594 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:36:17,670 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=2ms, max=119ms.
2024-01-18 11:36:39,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:36:40,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:37:17,727 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=2ms, max=119ms.
2024-01-18 11:37:39,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Synced up
2024-01-18 11:37:42,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:38:17,786 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=2ms, max=119ms.
2024-01-18 11:38:41,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:38:44,594 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:39:17,846 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=2ms, max=119ms.
2024-01-18 11:39:43,590 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:39:44,596 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Cleaned up
2024-01-18 11:40:17,901 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=2ms, max=119ms.
2024-01-18 11:40:43,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Synced up
2024-01-18 11:40:46,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:41:17,952 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=2ms, max=119ms.
2024-01-18 11:41:45,591 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2024-01-18 11:41:48,595 INFO pool-7-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2024-01-18 11:42:18,012 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (27 skipped) AgentAvroServlet: heartbeat processing stats: average=6ms, min=2ms, max=119ms.
2024-01-18 11:42:19,972 INFO SpringContextShutdownHook:com.cloudera.server.cmf.components.CmServerStateSynchronizer: CmServerStateSynchronizer is shutting down.
2024-01-18 11:42:19,987 INFO SpringContextShutdownHook:com.cloudera.server.cmf.components.CmServerStateSynchronizer: CM Server with id 7e8367e2-8df7-4ec0-a0ac-ba832b623750 removed from HA
2024-01-18 11:42:19,987 INFO SpringContextShutdownHook:com.cloudera.server.cmf.components.CmServerStateSynchronizer: CmServerStateSynchronizer has been stopped.
2024-01-18 11:42:19,987 INFO SpringContextShutdownHook:com.cloudera.server.cmf.components.ScmActive: ScmActive shutting down.
2024-01-18 11:42:19,987 INFO metric-schema-updater:com.cloudera.cmon.components.MetricSchemaManager: Breaking from sleep in schema update thread.
2024-01-18 11:42:21,446 INFO main:com.cloudera.server.cmf.Main: ================================================================================
2024-01-18 11:42:21,454 INFO main:com.cloudera.server.cmf.Main: Starting SCM Server. JVM Args: [-Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties, -Dfile.encoding=UTF-8, -Dcmf.root.logger=INFO,LOGFILE, -Dcmf.log.dir=/var/log/cloudera-scm-server, -Dcmf.log.file=cloudera-scm-server.log, -Dcmf.jetty.threshhold=WARN, -Dcmf.schema.dir=/opt/cloudera/cm/schema, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Dpython.home=/opt/cloudera/cm/python, -XX:+HeapDumpOnOutOfMemoryError, -Xmx4G, -XX:MaxPermSize=256m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/tmp, -Dcom.sun.management.jmxremote.ssl.enabled.protocols=TLSv1.2, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/sun.nio.cs=ALL-UNNAMED, --add-opens=java.base/java.lang=ALL-UNNAMED, --add-opens=java.base/java.lang.reflect=ALL-UNNAMED, --add-opens=java.desktop/java.awt.font=ALL-UNNAMED, --add-opens=java.base/java.util.regex=ALL-UNNAMED, --add-opens=java.base/java.util.zip=ALL-UNNAMED, --add-opens=java.base/java.util=ALL-UNNAMED, --add-opens=java.base/java.net=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.text=ALL-UNNAMED, --add-opens=java.base/javax.crypto=ALL-UNNAMED, -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2", -XX:OnOutOfMemoryError={CMF_PATH}/bin/killparent-cm-server], Args: [], Version: 7.9.5 (#36767575 built by jenkins on 20230119-0701 git: 4e4389c771898f43ef67f5847eb0d2f907d6ec91)
2024-01-18 11:42:22,103 INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.properties
2024-01-18 11:42:22,104 INFO main:com.cloudera.enterprise.CommonMain: Statistics not enabled, c3p0 JMX disabled
2024-01-18 11:42:23,448 INFO MLog-Init-Reporter:com.mchange.v2.log.MLog: MLog clients using slf4j logging.
2024-01-18 11:42:23,555 INFO main:com.mchange.v2.c3p0.C3P0Registry: Initializing c3p0-0.9.5.5 [built 11-December-2019 22:18:33 -0800; debug? true; trace: 10]
2024-01-18 11:42:23,599 INFO main:com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@e58db61d [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@5a5b1de9 [ acquireIncrement -> 3, acquireRetryAttempts -> 5, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, forceSynchronousCheckins -> false, identityToken -> 2vva03b17tbw56v1jwg9|7c7e73c5, idleConnectionTestPeriod -> 300, initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements -> 2500, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@59166477 [ description -> null, driverClass -> null, factoryClassLocation -> null, forceUseNamedDriverClass -> false, identityToken -> 2vva03b17tbw56v1jwg9|230a73f2, jdbcUrl -> jdbc:mysql://nitucc-vhm001.data.nitucc/scm?useUnicode=true&characterEncoding=UTF-8&useSSL=false, properties -> {password=******, user=******} ], preferredTestQuery -> null, privilegeSpawnedThreads -> false, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, extensions -> {}, factoryClassLocation -> null, identityToken -> 2vva03b17tbw56v1jwg9|25791d40, numHelperThreads -> 3 ]
2024-01-18 11:42:25,934 INFO main:com.cloudera.enterprise.CommonMain: Statistics not enabled, JMX will not be registered
2024-01-18 11:42:26,086 INFO main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: MYSQL database engine and table mapping: {InnoDB=[AUTH_ROLES, RELEASES_AUD, RELEASES, SERVICES, EXTERNAL_ACCOUNTS, AUTH_ROLES_SCOPES, HOSTS_AUD, HOST_TEMPLATE_TO_ROLE_CONF_GRP, DIAGNOSTICS_EVENTS, SCHEMA_VERSION, PARCEL_COMPONENTS, REPLICATION_METRICS, EXTERNAL_MAPPINGS, ROLE_CONFIG_GROUPS, CLUSTERS_AUD, SPRING_SESSION_ATTRIBUTES, CERTIFICATES, SERVICES_AUD, PROCESS_HEARTBEAT, COMMAND_SCHEDULES, SNAPSHOT_POLICIES, CLUSTER_MANAGED_RELEASES, USER_AUTH_ROLES, CONFIG_CONTAINERS, EXTERNAL_MAPPING_AUTH_ROLES, HOST_TEMPLATES, COMMANDS_DETAIL, CONFIGS, CLUSTER_UNDISTRIBUTED_RELEASES, ROLE_CONFIG_GROUPS_AUD, HOSTS, COMMANDS, CLUSTERS, ROLES, CLUSTER_ACTIVATED_RELEASES_AUD, SPRING_SESSION, USERS, CONTROL_PLANES, TAGS, HOST_HEARTBEAT, AUDITS, CLIENT_CONFIGS, EXTERNAL_ACCOUNTS_AUD, PROCESSES_DETAIL, USER_SETTINGS, DATA_CONTEXTS, CM_PEERS, CLUSTER_ACTIVATED_RELEASES, CREDENTIALS, ROLES_AUD, DATA_CONTEXT_TO_BASE_SERVICE, ROLE_STALENESS_STATUS, LOCKS, CONFIGS_AUD, REVISIONS, GLOBAL_SETTINGS, UPGRADE_STATE, STALENESS_DATA, METRICS, PROCESSES, PARCELS, USER_ROLES, CLIENT_CONFIGS_TO_HOSTS, CMSERVERS, CM_VERSION, CLIENT_CONFIG_HEARTBEAT, PROCESS_ACTIVE_RELEASES]}
2024-01-18 11:42:26,102 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.
2024-01-18 11:42:26,102 INFO main:com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 70700.
2024-01-18 11:42:26,408 ERROR main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Could not read license file /etc/cloudera-scm-server/license.txt
2024-01-18 11:42:26,414 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.
2024-01-18 11:42:26,414 INFO main:com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 70700.
2024-01-18 11:42:26,423 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.
2024-01-18 11:42:26,423 INFO main:com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 70700.
2024-01-18 11:42:26,424 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 70700
2024-01-18 11:42:26,459 INFO main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: ScmActive at bootup: Completed successfully.
2024-01-18 11:42:26,647 INFO main:com.cloudera.cmf.crypto.LicenseLoaderImpl: The license state of the product is TRIAL
2024-01-18 11:42:26,655 INFO main:com.cloudera.server.cmf.Main: CSP type is:none
2024-01-18 11:42:26,655 INFO main:com.cloudera.enterprise.AbstractWrappedEntityManager: Rolling back transaction that wasn't marked for rollback-only.
java.lang.Exception: Non-thrown exception for stack trace.
at com.cloudera.enterprise.AbstractWrappedEntityManager.close(AbstractWrappedEntityManager.java:163)
at com.cloudera.cmf.persist.CmfEntityManager.close(CmfEntityManager.java:398)
at com.cloudera.server.cmf.Main.initializeCSP(Main.java:457)
at com.cloudera.server.cmf.Main.<init>(Main.java:297)
at com.cloudera.server.cmf.Main.main(Main.java:262)
2024-01-18 11:42:26,928 INFO JmxPeriodicLogging-0:com.cloudera.enterprise.PeriodicJmx: JMX: {"beans":[{"name":"JMImplementation:type=MBeanServerDelegate","modelerType":"javax.management.MBeanServerDelegate","MBeanServerId":"nitucc-vad001.data.nitucc_1705574543492","SpecificationName":"Java Management Extensions","SpecificationVersion":"1.4","SpecificationVendor":"Oracle Corporation","ImplementationName":"JMX","ImplementationVersion":"11.0.11+9-LTS-194","ImplementationVendor":"Oracle Corporation"},{"name":"java.lang:type=Runtime","modelerType":"sun.management.RuntimeImpl","ClassPath":".:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/netty-transport-native-kqueue-4.1.48.Final.jar:lib/commons-httpclient-3.1.jar:lib/spring-webmvc-5.3.20.jar:lib/jackson-databind-2.13.4.jar:lib/kerby-config-1.0.0.jar:lib/log4j-api-2.17.1.jar:lib/jcip-annotations-1.0-1.jar:lib/commons-pool2-2.7.0.jar:lib/xercesImpl-2.12.2.jar:lib/jamonapi-2.73.jar:lib/async-http-client-2.12.1.jar:lib/jaxb-impl-2.4.0-b180830.0438.jar:lib/jackson-datatype-joda-2.13.4.jar:lib/metrics-core-2.1.3.jar:lib/poi-5.2.2.jar:lib/jetty-servlet-9.4.35.v20201120.jar:lib/parboiled-java-1.1.7.jar:lib/mx4j-3.0.1.jar:lib/jetty-io-9.4.35.v20201120.jar:lib/bcpg-jdk15on-1.61.jar:lib/spring-security-config-4.1.5.RELEASE.jar:lib/hsqldb-2.3.1.jar:lib/cxf-core-3.4.5.jar:lib/netty-common-4.1.48.Fin

Re: Getting Error while running the ansible play book for deploying CDP

Robin1986 — Fri, 19 Jan 2024 05:25:26 GMT

cloudera-scm-agent Error log

Cloudera-scm-server log.

Re: Getting Error while running the ansible play book for deploying CDP

james_jones — Sat, 20 Jan 2024 00:54:29 GMT

That's a lot of log. Some of the error messages you see are normal. I'm not sure what your issue is. Do you see Cloudera Management Service below the Cluster services in CM (at the very bottom when you click Cloudera Manager - top left)? If so, click Instances and figure out which components/roles are not started. You can also click and start them one by one. Then you can look at the startup logs in the CM UI pop-up after it starts or fails. Check in the order of STDOUT, STDERR and lastly ROLE LOG, which is the log after it is started. You may need to check the Full Log.