question Re: Tables regios stuck in RIT in Support Questions

Tables regios stuck in RIT

MrNicen — Sun, 05 May 2024 08:01:52 GMT

2024-05-04 15:27:08,590 WARN  [RpcServer.default.FPBQ.Fifo.handler=26,queue=1,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<SERVER>,16020,1714828981071, table=<NAMEAPACE>:<TABLE>, region=275bbfd7b7044afcba2deeeebf622b07 reported OPEN on server=l230-n2.<SERVER>,16020,1714828981071 but state has otherwise AND NO procedure is running

We have about 300 RIT and there alle stuck under the procedures & locks (<hbase master>:16010/procedures.jsp) we have some in state waiting_timeout.

We all ready tried restarting the cluster (hbase/zk) but got only whorse. There are 14 nodes in the cluster 3 are not assigned and we have a couple (about 4) that do all the work.

Where still on hbase 2.0.2 so we can't use hbck2 (2.0.3 >) and in hbase 2.0.2 hbck repair is disabled. Our average region count per server is about 330.

1. how do i get my regions unstuck?

Re: Tables regios stuck in RIT

VidyaSargur — Mon, 06 May 2024 08:29:21 GMT

@MrNicen, Welcome to our community! To help you get the best possible answer, I have tagged in our HBase experts @rki_ @smdas @SVB who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Tables regios stuck in RIT

rki_ — Mon, 06 May 2024 08:48:05 GMT

@MrNicen firstly, You will need to check the RS log of l230-n2.<SERVER>. As per the WARn, the region is already reported as OPEN by the same RS on which its trying again to open and going into a OPENING state.

It seems there are SCPs/ multiple assign procedures are running in the backend trying to open a region that is already open.

Considering that you are on a version that can neither use hbck2 jar nor the hbck1 commands, you can give try the below :

1) Stop both the Master

2) Move the contents of MasterProcWALs to a backup location

# hdfs dfs -mv /hbase/MasterProcWALs/* /tmp/

3) Start the Masters.

If the above doesn't solve the issue, I suggest raising a support case with Cloudera to review the logs.

Re: Tables regios stuck in RIT

Mr-nicen — Tue, 07 May 2024 09:01:17 GMT

Sorry wrong user.

Re: Tables regios stuck in RIT

MrNicen — Tue, 07 May 2024 08:54:15 GMT

We tried the solution and the number of RIT went down to about 134. but still not zero. Most of the RS log's are "clean". In de hbase master log we get a nasty.

2024-05-06 22:34:22,745 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=l230-n2.SERVER,16020,1715026584782, table=<NAMESPACE>:<TABLE>, region=e1b0976c6c5d759............

and

2024-05-06 22:26:25,230 WARN [RpcServer.default.FPBQ.Fifo.handler=33,queue=3,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.SERVER,16020,1715026585040, table=<NAMESPACE>:<TABLE>, region=d41e7923f187ee13d081591227c7805c reported OPEN on server=l225-n2.SERVER,16020,1715026585040 but state has otherwise AND NO procedure is running

these lines are repeated ever x sec's and are sets off 134 lines i guess the same as the number of stuck regions

I found out a can use hbck2 but have to add the "--skip version check option"

*Note had some trouble with my accounts, so resetted the wrong one.

Re: Tables regios stuck in RIT

rki_ — Tue, 07 May 2024 12:31:43 GMT

Try to bypass the stuck procedures using the hbck2 Jar and take the Region servers for a restart.

Re: Tables regios stuck in RIT

MrNicen — Tue, 07 May 2024 13:09:30 GMT

Together with some colleagues, we have analysed the logs after restarting HBase, and made the following lay mans analysis of the last startup log:

These seem innocent. HMaster starts en the RSs are not online yet:

2024-05-06 22:16:38,187 INFO [Thread-16] replication.ReplicationPeersZKImpl: Added peer cluster=l200-n4.<DOMAIN_M>,l201-n4.<DOMAIN_M>,l204-n4.<DOMAIN_M>:2181:/hbase-unsecure
2024-05-06 22:16:38,201 WARN [Thread-16] master.ServerManager: Server l224-n2.<DOMAIN_G>,16020,1659400236292 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n2.<DOMAIN_G>,16020,1659400236292]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l227-n2.<DOMAIN_G>,16020,1651336158836 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l227-n2.<DOMAIN_G>,16020,1651336158836]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n2.<DOMAIN_G>,16020,1713307724458 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n2.<DOMAIN_G>,16020,1713307724458]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l225-n1.<DOMAIN_G>,16020,1651396933254 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l225-n1.<DOMAIN_G>,16020,1651396933254]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l230-n2.<DOMAIN_G>,16020,1651396881056 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l230-n2.<DOMAIN_G>,16020,1651396881056]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n1.<DOMAIN_G>,16020,1660153454095 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n1.<DOMAIN_G>,16020,1660153454095]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l226-n2.<DOMAIN_G>,16020,1651397141435 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l226-n2.<DOMAIN_G>,16020,1651397141435]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l229-n1.<DOMAIN_G>,16020,1651336158866 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n1.<DOMAIN_G>,16020,1651336158866]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l228-n2.<DOMAIN_G>,16020,1651396801801 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l228-n2.<DOMAIN_G>,16020,1651396801801]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l227-n1.<DOMAIN_G>,16020,1651336158636 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l227-n1.<DOMAIN_G>,16020,1651336158636]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n1.<DOMAIN_G>,16020,1651336158334 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n1.<DOMAIN_G>,16020,1651336158334]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l226-n1.<DOMAIN_G>,16020,1651336158228 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l226-n1.<DOMAIN_G>,16020,1651336158228]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l229-n2.<DOMAIN_G>,16020,1651397094435 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n2.<DOMAIN_G>,16020,1651397094435]
2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l228-n1.<DOMAIN_G>,16020,1651336158675 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l228-n1.<DOMAIN_G>,16020,1651336158675]
2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l230-n1.<DOMAIN_G>,16020,1651336158117 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l230-n1.<DOMAIN_G>,16020,1651336158117]
2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l225-n2.<DOMAIN_G>,16020,1651336158291 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l225-n2.<DOMAIN_G>,16020,1651336158291]
2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l229-n1.<DOMAIN_G>,16020,1713308077605 is not currently online. Ignoring request to add it to draining list.
2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n1.<DOMAIN_G>,16020,1713308077605]

Here HMaster node l226-n3 becomes Active/primary and the SCPs are being started:


2024-05-06 22:16:38,203 INFO [Thread-16] master.HMaster: Active/primary master=l226-n3.<DOMAIN_G>,16000,1715026581275, sessionid=0x28f43c489256548, setting cluster-up flag (Was=true)
2024-05-06 22:16:38,282 INFO [Thread-16] master.HMaster: hbase:meta {1588230740 state=OPEN, ts=1715026596449, server=l225-n2.<DOMAIN_G>,16020,1714828981130}
2024-05-06 22:16:38,302 INFO [Thread-16] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1715026628302
2024-05-06 22:16:38,306 INFO [Thread-16] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=l228-n3.<DOMAIN_G>:2181,l229-n3.<DOMAIN_G>:2181,l230-n3.<DOMAIN_G>:2181
2024-05-06 22:16:38,306 INFO [Thread-16] zookeeper.ZooKeeper: Initiating client connection, connectString=l228-n3.<DOMAIN_G>:2181,l229-n3.<DOMAIN_G>:2181,l230-n3.<DOMAIN_G>:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@4e09d8ce
2024-05-06 22:16:38,316 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Opening socket connection to server l230-n3.<DOMAIN_G>/10.226.230.13:2181. Will not attempt to authenticate using SASL (unknown error)
2024-05-06 22:16:38,320 INFO [PEWorker-1] procedure.ServerCrashProcedure: Start pid=8, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l225-n2.<DOMAIN_G>,16020,1714828981130, splitWal=true, meta=true
2024-05-06 22:16:38,321 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.226.226.13:40378, server: l230-n3.<DOMAIN_G>/10.226.230.13:2181
2024-05-06 22:16:38,320 INFO [PEWorker-8] procedure.ServerCrashProcedure: Start pid=7, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l228-n1.<DOMAIN_G>,16020,1714828981415, splitWal=true, meta=false
2024-05-06 22:16:38,320 INFO [PEWorker-10] procedure.ServerCrashProcedure: Start pid=10, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l225-n1.<DOMAIN_G>,16020,1714829249747, splitWal=true, meta=false
2024-05-06 22:16:38,320 INFO [PEWorker-7] procedure.ServerCrashProcedure: Start pid=6, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l229-n2.<DOMAIN_G>,16020,1714828981440, splitWal=true, meta=false
2024-05-06 22:16:38,320 INFO [PEWorker-9] procedure.ServerCrashProcedure: Start pid=9, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l227-n2.<DOMAIN_G>,16020,1714829249989, splitWal=true, meta=false
2024-05-06 22:16:38,320 INFO [PEWorker-6] procedure.ServerCrashProcedure: Start pid=5, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l224-n1.<DOMAIN_G>,16020,1714828981244, splitWal=true, meta=false
2024-05-06 22:16:38,320 INFO [PEWorker-4] procedure.ServerCrashProcedure: Start pid=3, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l228-n2.<DOMAIN_G>,16020,1714828981422, splitWal=true, meta=false
2024-05-06 22:16:38,320 INFO [PEWorker-5] procedure.ServerCrashProcedure: Start pid=4, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l226-n2.<DOMAIN_G>,16020,1714828981235, splitWal=true, meta=false
2024-05-06 22:16:38,320 INFO [PEWorker-2] procedure.ServerCrashProcedure: Start pid=1, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l224-n2.<DOMAIN_G>,16020,1714828981192, splitWal=true, meta=false
2024-05-06 22:16:38,321 INFO [PEWorker-12] procedure.ServerCrashProcedure: Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l230-n2.<DOMAIN_G>,16020,1714828981071, splitWal=true, meta=false
2024-05-06 22:16:38,321 INFO [PEWorker-13] procedure.ServerCrashProcedure: Start pid=13, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l230-n1.<DOMAIN_G>,16020,1714828981173, splitWal=true, meta=false
2024-05-06 22:16:38,321 INFO [PEWorker-11] procedure.ServerCrashProcedure: Start pid=11, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l226-n1.<DOMAIN_G>,16020,1714829249771, splitWal=true, meta=false
2024-05-06 22:16:38,321 INFO [PEWorker-14] procedure.ServerCrashProcedure: Start pid=14, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l227-n1.<DOMAIN_G>,16020,1714828981420, splitWal=true, meta=false
2024-05-06 22:16:38,321 INFO [PEWorker-3] procedure.ServerCrashProcedure: Start pid=2, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l229-n1.<DOMAIN_G>,16020,1714828981427, splitWal=true, meta=false

Zookeeper connection:

2024-05-06 22:16:38,326 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Session establishment complete on server l230-n3.<DOMAIN_G>/10.226.230.13:2181, sessionid = 0x38f43c48945656c, negotiated timeout = 60000

House keeping of WALs in Thread-16

2024-05-06 22:16:38,327 INFO [Thread-16] cleaner.CleanerChore: Cleaner pool size is 1
2024-05-06 22:16:38,328 INFO [Thread-16] cleaner.LogCleaner: Creating OldWALs cleaners with size=2

In PEWorker-9 more WAL management specifically related to l225-n2 (Why?):

2024-05-06 22:16:38,362 INFO [PEWorker-9] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting is empty dir, no logs to split
2024-05-06 22:16:38,362 INFO [PEWorker-9] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting] for [l225-n2.<DOMAIN_G>,16020,1714828981130]
2024-05-06 22:16:38,365 INFO [PEWorker-9] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting] in 3ms

In PEWorker-9 initialisation of the AssignProcedure

2024-05-06 22:16:38,397 INFO [PEWorker-9] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}]

Zookeeper Replication Peers, since we have replication enabled

2024-05-06 22:16:38,403 INFO [Thread-16] replication.ReplicationPeersZKImpl: Added peer cluster=l200-n4.<DOMAIN_M>,l201-n4.<DOMAIN_M>,l204-n4.<DOMAIN_M>:2181:/hbase-unsecure

Still waiting for the first RS to call in:

2024-05-06 22:16:38,404 INFO [Thread-16] master.HMaster: Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1715026584479, completionTime=-1
2024-05-06 22:16:38,404 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=0; waited=0ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms

Re: Tables regios stuck in RIT

MrNicen — Tue, 07 May 2024 20:41:41 GMT

Somehow it was decided that for table hbase:meta the region should enter state RUNNABLE on RS l225-n2.
Why l225-n2? Note, rit is still reported offline.

2024-05-06 22:16:38,408 INFO  [PEWorker-14] procedure.MasterProcedureScheduler: Took xlock for pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740
2024-05-06 22:16:38,417 INFO  [PEWorker-14] assignment.AssignProcedure: Starting pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=l225-n2.<DOMAIN_G>,16020,1714828981130; forceNewPlan=false, retain=true target svr=null

Here l227-n2 is calling in and Thread-16 reacts.

2024-05-06 22:16:38,536 INFO  [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l227-n2.<DOMAIN_G>,16020,1715026585628
2024-05-06 22:16:38,555 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=1; waited=151ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms

The following may be the first relevant WARN. Its is about ServerStateNode for l227-n2 being empty. Are these nodes persisted in Zookeeper?

2024-05-06 22:16:38,570 WARN  [PEWorker-10] assignment.AssignmentManager: Why is ServerStateNode for l227-n2.<DOMAIN_G>,16020,1715026585628 empty at this point? Creating...
2024-05-06 22:16:38,570 INFO  [PEWorker-10] assignment.AssignProcedure: Early suspend! pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=l227-n2.<DOMAIN_G>,16020,1715026585628

Next, all RSs are registering themselves and ephemeral node's are being created (I think in in ZK):

2024-05-06 22:16:38,579 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l227-n2.<DOMAIN_G>,16020,1715026585628]
2024-05-06 22:16:39,409 INFO  [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l228-n1.<DOMAIN_G>,16020,1715026585682
2024-05-06 22:16:39,432 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l228-n1.<DOMAIN_G>,16020,1715026585682]
2024-05-06 22:16:39,457 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=2; waited=1053ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
2024-05-06 22:16:40,100 INFO  [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l224-n2.<DOMAIN_G>,16020,1715026584653
2024-05-06 22:16:40,108 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=3; waited=1704ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
2024-05-06 22:16:40,112 INFO  [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l230-n2.<DOMAIN_G>,16020,1715026584782
2024-05-06 22:16:40,115 INFO  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l226-n1.<DOMAIN_G>,16020,1715026584512
2024-05-06 22:16:40,116 INFO  [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l226-n2.<DOMAIN_G>,16020,1715026584780
2024-05-06 22:16:40,120 INFO  [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l230-n1.<DOMAIN_G>,16020,1715026584638
2024-05-06 22:16:40,132 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l224-n2.<DOMAIN_G>,16020,1715026584653]
2024-05-06 22:16:40,135 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l230-n2.<DOMAIN_G>,16020,1715026584782]
2024-05-06 22:16:40,135 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l226-n1.<DOMAIN_G>,16020,1715026584512]
2024-05-06 22:16:40,135 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l226-n2.<DOMAIN_G>,16020,1715026584780]
2024-05-06 22:16:40,142 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l230-n1.<DOMAIN_G>,16020,1715026584638]
2024-05-06 22:16:40,154 INFO  [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l225-n1.<DOMAIN_G>,16020,1715026584877
2024-05-06 22:16:40,158 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=8; waited=1754ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
2024-05-06 22:16:40,175 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l225-n1.<DOMAIN_G>,16020,1715026584877]
2024-05-06 22:16:40,323 INFO  [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l224-n1.<DOMAIN_G>,16020,1715026584891
2024-05-06 22:16:40,343 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l224-n1.<DOMAIN_G>,16020,1715026584891]
2024-05-06 22:16:40,359 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=9; waited=1955ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
2024-05-06 22:16:40,528 INFO  [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l225-n2.<DOMAIN_G>,16020,1715026585040
2024-05-06 22:16:40,529 INFO  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l229-n1.<DOMAIN_G>,16020,1715026584835
2024-05-06 22:16:40,531 INFO  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l229-n2.<DOMAIN_G>,16020,1715026584960
2024-05-06 22:16:40,549 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l225-n2.<DOMAIN_G>,16020,1715026585040]
2024-05-06 22:16:40,551 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l229-n1.<DOMAIN_G>,16020,1715026584835]
2024-05-06 22:16:40,552 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l229-n2.<DOMAIN_G>,16020,1715026584960]
2024-05-06 22:16:40,559 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=12; waited=2155ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
2024-05-06 22:16:40,840 INFO  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l227-n1.<DOMAIN_G>,16020,1715026585107
2024-05-06 22:16:40,861 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=13; waited=2457ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
2024-05-06 22:16:40,864 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l227-n1.<DOMAIN_G>,16020,1715026585107]
2024-05-06 22:16:41,088 INFO  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l228-n2.<DOMAIN_G>,16020,1715026585349
2024-05-06 22:16:41,112 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=2707ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
2024-05-06 22:16:41,112 INFO  [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l228-n2.<DOMAIN_G>,16020,1715026585349]

Hold on, we are almost reaching the point where the **bleep** hits the fan. The region of htable hbase:meta is dispatched to l227-n2, no mention anymore of rit=offline. But note that the RS has changed from l225-n2 to l227-n2, replicaid=0 in Zookeeper

2024-05-06 22:16:42,579 INFO  [PEWorker-7] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2.<DOMAIN_G>,16020,1715026585628
2024-05-06 22:16:42,585 INFO  [PEWorker-7] assignment.RegionTransitionProcedure: Dispatch pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740

Server manager is awaiting even more RSs, but we have 14 and they are all registered at this point.

2024-05-06 22:16:42,615 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=4211ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-1504ms

Now from PEWorker-8 Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2 and finally the ProcedureExecutor reports state=SUCCESS for AssignProcedure table=hbase:meta:

2024-05-06 22:16:43,572 INFO  [PEWorker-8] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2.<DOMAIN_G>,16020,1715026585628
2024-05-06 22:16:43,583 INFO  [PEWorker-8] procedure2.ProcedureExecutor: Finished subprocedure pid=15, resume processing parent pid=8, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=l225-n2.<DOMAIN_G>,16020,1714828981130, splitWal=true, meta=true
2024-05-06 22:16:43,584 INFO  [PEWorker-8] procedure2.ProcedureExecutor: Finished pid=15, ppid=8, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 5.1780sec, unfinishedSiblingCount=0

Time ticks on and no more region server registrations:

2024-05-06 22:16:44,119 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=5714ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-3007ms
2024-05-06 22:16:45,623 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=7219ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-4512ms

No we arrive at the point where the problem manifests itself:

2024-05-06 22:16:45,644 INFO  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740
2024-05-06 22:16:45,644 WARN  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628

From her on I am skipping the INFO logs of the ServerManager Waiting on regionserver count=14. More WARNs:


2024-05-06 22:16:48,650 WARN  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628
2024-05-06 22:16:51,655 INFO  [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740
...

Several similar errors

Than we arrive at master is running:

2024-05-06 22:17:08,175 INFO  [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=29771ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-27064ms
2024-05-06 22:17:08,426 INFO  [Thread-16] master.ServerManager: Finished waiting on RegionServer count=14; waited=30021ms, expected min=1 server(s), max=NO_LIMIT server(s), master is running

Next we get loads these regions being assigend:

2024-05-06 22:17:08,614 INFO  [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=bd9b9493492af8b42a99ff7e4cee8eb8, regionState=OPEN, lastHost=l230-n2.<DOMAIN_G>,16020,1714828981071, regionLocation=l230-n2.<DOMAIN_G>,16020,1714828981071, openSeqNum=531452
2024-05-06 22:17:08,615 INFO  [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=de38ce1846144c1eb843ca9248efb791, regionState=OPEN, lastHost=l225-n2.<DOMAIN_G>,16020,1714828981130, regionLocation=l225-n2.<DOMAIN_G>,16020,1714828981130, openSeqNum=516441
2024-05-06 22:17:08,615 INFO  [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=6546461582f9e25f18f5c77f6cf8fe6d, regionState=OPEN, lastHost=l225-n2.<DOMAIN_G>,16020,1714828981130, regionLocation=l225-n2.<DOMAIN_G>,16020,1714828981130, openSeqNum=403570
...

Re: Tables regios stuck in RIT

MrNicen — Tue, 07 May 2024 20:47:33 GMT

The following ones are odd:

2024-05-06 22:17:08,626 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=audit:dzs, region=3da7e18863977749267f48437295889d

...

2024-05-06 22:17:09,760 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_13>:<TABLE_46>,
region=0486a586e762a8fc837ad4cf3f48c0a1

...

2024-05-06 22:17:09,763 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l227-n1.<DOMAIN_G>,16020,1714828981420, table=<NAMESPACE_13>:<TABLE_63>, region=8b12508d2ef09a011875540b383135dc

...

2024-05-06 22:17:09,931 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l228-n1.<DOMAIN_G>,16020,1714828981415, table=<NAMESPACE_17>:<TABLE_46>, region=fcf92fdae49288267f5b94f80bafd186

...

2024-05-06 22:17:10,046 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l227-n1.<DOMAIN_G>,16020,1714828981420, table=<NAMESPACE_19>:<TABLE_12>, region=7766e1e93ed788e0233d1286968fc132

...

2024-05-06 22:17:10,046 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_19>:<TABLE_12>, region=969d766f8ecaf8074572a97dd7a4ad89

...

2024-05-06 22:17:10,047 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=<NAMESPACE_19>:<TABLE_12>, region=2d84f975f555ee262c64618055075ccf

...

2024-05-06 22:17:10,047 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l229-n2.<DOMAIN_G>,16020,1714828981440, table=<NAMESPACE_19>:<TABLE_12>, region=2f1f50ee36b7d30b9f234380a02cd154

2024-05-06 22:17:10,081 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=<NAMESPACE_19>:<TABLE_41>, region=dcf0f06facb516b3a05803f1510d35ea

2024-05-06 22:17:10,082 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l228-n1.<DOMAIN_G>,16020,1714828981415, table=<NAMESPACE_19>:<TABLE_41>, region=ba1f621709cf61c2c7ac523765f11a2d

2024-05-06 22:17:10,214 INFO  [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_21>:<TABLE_41>, region=38e9a26a5df320c95349afc7ca374604

These keep repeating

2024-05-06 22:17:09,691 INFO  [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740
2024-05-06 22:17:09,691 WARN  [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628

After assignment.RegionStateStore finishes this first round of assignments in [Thread-16]:

2024-05-06 22:17:11,444 INFO  [Thread-16] assignment.AssignmentManager: Number of RegionServers=14
2024-05-06 22:17:11,459 INFO  [Thread-16] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1715026691459
2024-05-06 22:17:11,460 INFO  [Thread-16] assignment.AssignmentManager: Joined the cluster in 3.0330sec

Next, we arrive at messages from master.SplitLogManager

2024-05-06 22:17:11,656 INFO  [PEWorker-1] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting is empty dir, no logs to split
2024-05-06 22:17:11,661 INFO  [PEWorker-1] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting] for [l229-n1.<DOMAIN_G>,16020,1714828981427]
2024-05-06 22:17:11,663 INFO  [PEWorker-4] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting is empty dir, no logs to split
2024-05-06 22:17:11,663 INFO  [PEWorker-4] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting] for [l227-n1.<DOMAIN_G>,16020,1714828981420]
2024-05-06 22:17:11,664 INFO  [PEWorker-1] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting] in 3ms
2024-05-06 22:17:11,666 INFO  [PEWorker-4] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting] in 3ms
2024-05-06 22:17:11,667 INFO  [PEWorker-16] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting is empty dir, no logs to split
2024-05-06 22:17:11,667 INFO  [PEWorker-16] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting] for [l224-n1.<DOMAIN_G>,16020,1714828981244]
2024-05-06 22:17:11,669 INFO  [PEWorker-16] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting] in 2ms
2024-05-06 22:17:11,690 INFO  [PEWorker-11] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting is empty dir, no logs to split
2024-05-06 22:17:11,690 INFO  [PEWorker-11] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting] for [l226-n1.<DOMAIN_G>,16020,1714829249771]
2024-05-06 22:17:11,693 INFO  [PEWorker-11] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting] in 3ms
...

and after this follow the long lines from the procedure2.ProcedureExecutor

2024-05-06 22:17:11,837 INFO  [PEWorker-16] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79}, {pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9}, {pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03}, {pid=19, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:zbl_persistent, region=ba37bf35a4430884d1a6ebbca95a8387}, {pid=20, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:grh_persistent, region=68531e269838537d534e81683e2a7097}, {pid=21, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_28>, region=72f61b803e206cc95dc2a05b1112d39f}, {pid=22, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_5>, region=f696c18f229b7b6525405d55afdd57cb}, {pid=23, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:grh_client_timeline, region=285ddc10b5d812318af616f7986ac6d1}, {pid=24, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_51>, region=f56d6589a0660ea69432e3ff1bfa6941}, {pid=25, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=0a0f5b07562c6220f4b15c8128748925}, {pid=26, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_15>, region=d4568f96d953bd370840c8942c263d6d}, {pid=27, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_12>, region=7b8c2748761fe9da697b565bd51c29ff}, {pid=28, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_13>, region=911abd94189cd5cacad87054333c7756}, {pid=29, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=d028ce2dae91e85c853dfbf24167b89f}, {pid=30, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=907ba360d0a32358465ffa19b877d0ff}, {pid=31, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_6>, region=07c3225b0f58be3adfe3e10f4054e5e8}, {pid=32, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_31>, region=eba2fd53661e007937008e1b4b8723a2}, {pid=33, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=96d4c009fb28c87f11371273343c86c2}, {pid=34, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_53>, region=a1cc9bb5815382500ebc1f8ff2fe2889}, {pid=35, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_45>, region=72af38e5137228c4a96c5afd1483141b}, {pid=36, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_72>, region=2ae51a84d97bc465eca9b9d534585752}, {pid=37, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:tbb_persistent, region=60f28a92e81332855bc0bc8814b43c9c}, {pid=38, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=1041abc1f0324a01072e30ac4651068f}, {pid=39, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:grh, region=913fc5fbecc382c9eb130bbca997a52a}, {pid=40, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:idp, region=52b26bbcb3470e2fda96a27cfb2bc728}, {pid=41, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:nbr, region=9d53deeb6951b5625c714f52f40cc2e2}, {pid=42, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:ztw, region=3204e2b67fcf1ff91580c11f919c9261}, {pid=43, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_32>, region=f9459615516226d166054c69348c614c}, {pid=44, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_46>, region=cb9fe991b45b07aaca74b1b19cd1d8a9}, {pid=45, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_24>, region=3c0ac9e61eb863ac6834645a45b2ce92}, {pid=46, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>: Employee, region=3152b0b94f3dab7872292d320681192f}, {pid=47, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_29>, region=2c1fabd4bedd491f1d74f814ca788c24}, {pid=48, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=698bf8a9ad8970d1059273622b3beea9}, {pid=49, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=7a54ab62ca00389083cacfd18859b19b}, {pid=50, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:ijs_persistent, region=9ce8fccc5665f0256fb8ab8e25618bcc}, {pid=51, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:ijs_persistent, region=fa3effb1e5c14db984279936ab111895}, {pid=52, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=53d819a79ac851e839f5ff6a6b149b19}, {pid=53, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_15>, region=5864381b7aedceae4ffae2d1eb84620e}, {pid=54, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_51>, region=39dc5b21757fb3745c703656d2385132}, {pid=55, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_63>, region=484aacabb3202a69d985c8316db6ed2e}, {pid=56, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=5aa11c337c3f916a35a27dea488e22d6}, {pid=57, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=971001559dbb1da0c4518773a5bd598f}, {pid=58, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_72>, region=7d8e127283f4c0116ac780ad7274a419}, {pid=59, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_33>, region=145a675885029dd31097cbd697243703}, {pid=60, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_41>, region=4639b130e351fc822f26948d88ee50e1}, {pid=61, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=3e29eed2edc1b2abb1dbe7e416087835}, {pid=62, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_7>, region=815ce30ee31cc91f4a6ae4b1ff787bad}] 2024-05-06 22:17:11,885 INFO  [PEWorker-11] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=63, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:wlt_by_query, region=85d79840d75bd9d605e7f3fa9a330655}, {pid=64, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:wlt_quantifier_index, region=9372d2b0f3629ed5c088dfc0aa541961}]*no* further _formatting_ is done here

And first procedures reporting results?

2024-05-06 22:17:11,887 INFO  [PEWorker-8] procedure.MasterProcedureScheduler: Took xlock for pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9
2024-05-06 22:17:11,889 INFO  [PEWorker-7] procedure.MasterProcedureScheduler: Took xlock for pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79

Re: Tables regios stuck in RIT

MrNicen — Tue, 07 May 2024 21:04:38 GMT

More long ones:

2024-05-06 22:17:11,944 INFO [PEWorker-14] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=65, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_38>, region=d26d9da451ba6ee5d917d0e135f5a7bf}, {pid=66, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_2>, region=ac46967db9dfca5c91369164ac1c9469}, {pid=67, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=8ea10f9305a566d8efadbfcb2c6d87de}, {pid=68, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_37>, region=c038bc8eadb331b104131184d88316ce}, {pid=69, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=d2df8e2457a319244cfa85d898773f53}, {pid=70, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=c32fbee54febb51b1278c139729c9991}, {pid=71, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_61>, region=85a1c4f8c9303a693d2b0baa3f04e576}, {pid=72, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_43>, region=cf3a8c84ee842791b876765071465d98}, {pid=73, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_26>, region=0bb135123341acc459bc2b0e095c571d}, {pid=74, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_61>, region=60936d1899edbaae0788c2d928e34be0}, {pid=75, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=c3763274a6c84f17cafbb831c1a81132}, {pid=76, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=9f4bb127925ef66b2e2f181ee0b69d02}, {pid=77, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_14>, region=846e8ddaec0fffcb97d44480c7cc8798}, {pid=78, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_63>, region=cb50cf460205d12c5bee0e2fb1f4c21c}, {pid=79, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_73>, region=09b701e10fd060bf0901ff8ce55778e4}, {pid=80, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=a228e68c6d231c7d37a15c1c048bef58}, {pid=81, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_17>, region=7ba59a23baf79edc23ebbc5b152936a3}, {pid=82, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_6>, region=e94e891f60a0a0c194700f73504ab4a1}, {pid=83, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_54>, region=be8f1361b364a21f20300aee3c174464}, {pid=84, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_43>, region=1ee1d509aa4c312c6d48e2bcce1d14a2}, {pid=85, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=1621fa407abeeb4c112638af74553342}, {pid=86, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=530c9d659a0416d0f442a09e8070d106}, {pid=87, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=bd644295d91e525001cb1c08f82c06cb}, {pid=88, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_46>, region=69c55317c0eaec7c118fa09308c34997}, {pid=89, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_41>, region=a882652e5d221f3fc56efc5214970553}, {pid=90, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_25>, region=cf45f56e80d4b860220db6589e55fa4b}, {pid=91, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=f6f18f4f98aae325d3aab52560128a2c}, {pid=92, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=bc4a3997c5f8dd993dd3079044bb9443}, {pid=93, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_51>, region=ab6f3e8d44332b4e78a0aae110b26603}, {pid=94, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_29>, region=a9ed6a68f9edc2c93bca66f146d24e85}, {pid=95, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_14>, region=358157917db43e1aab7c64bde66e11b8}, {pid=96, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_26>, region=1491a0474a21f04733065b84a7ea0690}, {pid=97, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_42>, region=143bd5d9cd709b8b346bb166bbce71c2}, {pid=98, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_44>, region=e5eba35899c2e90162b869d8e75a0c76}]
2024-05-06 22:17:11,950 INFO [PEWorker-6] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=99, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_27>, region=fd8364d6adfd11711610317f468a518a}, {pid=100, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_21>, region=81be87773b6c3ffacc0a07c9da8d7dd4}, {pid=101, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=ec8d8cf7f7aaa6e9740a1c43a2942ff7}, {pid=102, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_39>, region=31434aa06033a0edf3cb394a7d64a15d}, {pid=103, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_46>, region=b78b117f1426c5ce0e528eb51ad3ec07}, {pid=104, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_9>, region=fe81d1f92167a6099eb5516668314e3a}, {pid=105, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_62>, region=7b1b806d6b9fcfe5ac2f1d54a7a6229a}, {pid=106, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_1>, region=5b4d78b09c99ac59d10c83fff9303cd0}, {pid=107, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_6>, region=40c174d887cbfab7da2ba1977a60fe94}, {pid=108, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_26>, region=f1bf3decd899563212c071e24508b97d}, {pid=109, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_timespan_index, region=8f794f8b6ac4f70770d0f80a5e9b02ce}, {pid=110, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=3609823c9e0c2bc9c7d65d716e290bb7}, {pid=111, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_23>, region=d004e125f9dfe02fbfc8df621d4c9878}, {pid=112, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_36>, region=a724e0984e13a5dc9160a68ab72f5e29}, {pid=113, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=ae445ff6e5c92bed212429c28bf2795d}, {pid=114, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_31>, region=5c169953e27e09c477c78bfefe9ed88a}, {pid=115, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_39>, region=5c512440cca3566741e6d53e3bb29eb1}, {pid=116, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_27>, region=cab78895641204b76abb7229418ac4d2}, {pid=117, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_reference, region=dd6daf6ff096794f8ebddc0d28049b6e}, {pid=118, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=36add998cd3ce2524203ac9d69906530}, {pid=119, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=587fec8406b9994f1981094ef7e7ebda}, {pid=120, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_54>, region=b92a93a355d6a1ad7ff11bf3e34fb9ee}, {pid=121, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_71>, region=ccc28d5406e44135221ce70fa580cb6b}, {pid=122, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_18>, region=f315381dc89c081c0ff999106ebd0f9e}, {pid=123, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_7>, region=3f3e045bdedf9708c9167aef64477bca}, {pid=124, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=6898c984f4ef6345ff9656f783d0decf}, {pid=125, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=85a72fc500b1d803a1c20e7dd312994a}, {pid=126, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=85df5947445cfa615d210549d4113544}, {pid=127, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=08a4d55e7f60d570e90e08bca932e274}, {pid=128, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=5e6e5a2254aaaa9cd6bcc07787bd7b68}, {pid=129, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=1ff9f8b3fbdd559236bfbf3341a881db}, {pid=130, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=ad8cd76e0a891016efa454b052a4d709}, {pid=131, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=4666ecb9e11eeb2a1a92a5db20e9fc77}, {pid=132, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5a47edc1544c02826a522a8806af7097}, {pid=133, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:wlt, region=5eb06b8067fdbccef433aa61685e2bf1}, {pid=134, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=358a62174fa70ce3e594ac48b65265da}, {pid=135, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_52>, region=a7c63347b3eb5dde263cbfdd65b77a69}, {pid=136, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=f9f52135ba20e1fe76cf371acab6bd00}, {pid=137, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=29431818e46baa2963d67e2635ee83dc}, {pid=138, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_27>:message-bus, region=7da31638b73cf866e7995b1c77cfb2a4}, {pid=139, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_46>, region=0ff03da8b9c98538bbe04d1512144c76}, {pid=140, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=275807cbabad3122b0baddbc1d7352f7}, {pid=141, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_57>, region=887d0888424b0e60060a9cc8659bd35f}, {pid=142, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=987b0dc7408d25aa411924236432d746}, {pid=143, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=f14a6fb3f90f81ee77388964344c2eba}, {pid=144, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=6e505ffbb8a41fd3ec5ffa5678063228}, {pid=145, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_60>, region=14ec3ec9cfd01b8172aac7c2b6f99a6d}, {pid=146, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_59>, region=9b9897f6f40f973796471ab12a7ba9ec}]
2024-05-06 22:17:11,954 INFO [PEWorker-10] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=147, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=00d20f4dd79bcebe1ecfc93b370d1941}, {pid=148, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=b3f91f5c8ef858e6d15c5bb8f7a7cd26}, {pid=149, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:cds, region=ac189d86bf78b4b9a24c476663874eb7}, {pid=150, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe, region=606e3055bcdd795113d6fa794e1d9843}, {pid=151, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_13>, region=2fe2e69dc40e61d76960e2fb9d87731f}, {pid=152, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_46>, region=c78535817a6d3956701a99e89baf6ca7}, {pid=153, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=0cb990ba90c53422b422e6a2b172d4ff}, {pid=154, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_19>, region=1c5466966c1aa44b2191db632806e4d5}, {pid=155, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_10>, region=71c761063de6a50e16750ab15921dabd}, {pid=156, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=5234265075f572118fd896d57b27018c}, {pid=157, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:hgg_client_timeline, region=753d0f68de8e47864ecdf59bc6c3f084}, {pid=158, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_persistent, region=ebc1328fab65efd0d9314130ceac2b94}, {pid=159, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_by_query, region=8f4c816f5ad863cd7176defab9d8b979}, {pid=160, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_by_query, region=272a0150d5fb7edb81e1230a5d38db18}, {pid=161, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_46>, region=4dd22258f7282411a06566357e44e335}, {pid=162, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_43>, region=5344ff13fb3db964fe346099ccbb5ad4}, {pid=163, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_40>, region=e762e13ed49b02ae79ced188bb68c6cb}, {pid=164, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=4f8445c40820a61633833e2e028d7441}, {pid=165, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_28>, region=c1ed331e961eeeb2bb6fa9d50f73a040}, {pid=166, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=0e3f779f3590717ee562fc40ac26faf2}, {pid=167, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0133fc91fc6d7bef529682456d19a774}, {pid=168, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=9755e4208ced1b938c53017ed30b6c12}, {pid=169, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=cfafb084581d983eef636cd1c09be90d}, {pid=170, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_46>, region=09af2afbe817abcf95dc0cb8825ed9fd}, {pid=171, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_by_query, region=6ff424adf916a4bdfbc4c58ad897d3fc}, {pid=172, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_by_query, region=9c4c92c7246ca209d7a02dca9033f544}, {pid=173, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_27>, region=432de3f682b58a0745464c9335be837e}, {pid=174, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=334b11e67ce7f61b8165e14ad470c9cf}, {pid=175, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_1>, region=9e7c8d1219f6b4f094f5d372aee740c3}, {pid=176, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=aec330cf8c9899290b1cca3d8dbaf7f5}, {pid=177, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=ccd9e06e5f8fcfe9dde1b65133fe8839}, {pid=178, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=9c1eec8741efe2d89c0567484cd638f0}, {pid=179, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=1bbc996cfa734e0fc6c70bacfa5aa4da}, {pid=180, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=7b9fa4da04f481d3258d7fd57b72a354}, {pid=181, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=8e894c6a576efb759fba7cc4da6be1df}, {pid=182, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5cd6925ee92449240ce14a37b275330f}, {pid=183, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_37>, region=48231aab340dff0874a068d75e53fa5e}, {pid=184, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_26>, region=3608dbe215817b2047c73be91604bfdf}, {pid=185, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_by_query, region=5c874fc02cd78b2a2da75bb156789c4e}, {pid=186, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_by_query, region=47211257f5f82a304ffaf05a8942858a}, {pid=187, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_48>, region=b0b022264b64be0b9fe4dc849e1cf1b5}, {pid=188, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_3>, region=01f9fd1bb65e07860fa74ed03266ac9c}, {pid=189, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_quantifier_index, region=2c017dc1457c9390388cd5644d3e3569}, {pid=190, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=bb88b7509375a9e1caa80b2c75d42a6b}, {pid=191, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=7a1d6e6b9390f9be391a2aa246282557}, {pid=192, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=35ae73ec75648c6943cffa7fbfcf42f4}, {pid=193, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_33>, region=adf0fc266fef938f1fb10efb21580df9}, {pid=194, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=f19bade589507b3931911a7a99c2f264}, {pid=195, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_13>, region=c53bd3f2a33d92f039ba4bb16efa518c}, {pid=196, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_58>, region=f3ceda41c170dc8ea5d6b2ef1a057b62}, {pid=197, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=769229ad8d8e8b8498974d2a643ea10b}, {pid=198, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=58ea387d639ee0f764ba90310584c9d4}, {pid=199, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_54>, region=23e746b75b557c89f5f3bb3f0d0de4b2}, {pid=200, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_11>, region=48e2a9ad25715bf409c0ab8390f92144}]
2024-05-06 22:17:11,966 INFO [PEWorker-13] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=201, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_15>, region=b5c4d477bba1dfbdcc2165c523bd7395}, {pid=202, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat, region=bbbc7f79a39a1c8a417a0a7f9054d1a7}, {pid=203, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_60>, region=09cdd7faf2a89a737d813d648e1b028f}, {pid=204, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_62>, region=18a0f41c5988a80a493e05cae42ddf0c}, {pid=205, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_25>, region=bcbdc1f2e315acd61ec815d69df8881a}, {pid=206, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=7ccd5f985123343cd540a4cbdc9a704d}, {pid=207, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_7>, region=b6031e76568526723101704dd52bfc3e}, {pid=208, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_9>, region=c54357dec4f34c820a50ca53852508f5}, {pid=209, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=d53818a4613c3655c8809b488dd13fe5}, {pid=210, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=1c56709a7ef1202b2ca3745e178c1ab3}, {pid=211, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_71>, region=4cc6da9099a4fb0cb217a1dba51ef6bd}, {pid=212, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_37>, region=a7f97c7aabc91f42af675c73f03b3546}, {pid=213, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_35>, region=463c29108637a39f2429745d39ff73e7}, {pid=214, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=25ac0b75eae580989551c7d8b3773732}, {pid=215, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=3bc0c1a61c45f68dcbd0af9fb496e7b4}, {pid=216, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_18>, region=e1a8fb3db5e9daa5e2f32d10a29046ca}, {pid=217, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_46>, region=b912321cbb0dcea3280a6015465d5f78}, {pid=218, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_21>, region=87cf47f804e40b1a52a77f5f39b3f184}, {pid=219, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=e17d2f7c9838590707239cc949ec0eeb}, {pid=220, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=4699b0b7032fde3b1736d5d445091add}, {pid=221, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=9fe9190ef7db10a90c2dcf62187eb737}, {pid=222, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_55>, region=f17c894285999f568dc2d13307c30f32}, {pid=223, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_50>, region=59b8a67b8af82dfbb08afa40e6245ec8}, {pid=224, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=be9a5a341a519c132b4daddbd9cd5804}, {pid=225, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=88fa6bfdc0ef4c45b1fe5eb22de6e6aa}, {pid=226, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=d4b53b21efe1bf9a4632dced64ee0d1a}, {pid=227, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=9abbedce0ecb7e64aaab441dfb867922}, {pid=228, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=7137884e41ce4bd388567ccd7c91ad9e}, {pid=229, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=b03a1ac196a2758819ab14bcc2d182b6}, {pid=230, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=abc3172931d7f48c6613a6cf8268ed5b}, {pid=231, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=265d2fedcc86f4101420ff639be21383}, {pid=232, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_46>, region=15d526f60ea517e2129bc373e63ddf11}, {pid=233, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_72>, region=4a074ba4dca8703f0a7ddd87fcd50bfe}, {pid=234, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_47>, region=f494d0b158b7c3dd986314e242d93770}, {pid=235, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_32>, region=3301fea1f596540b4c605a8afb2a83a6}, {pid=236, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_40>, region=d3160c06c71d5ec702d391222f21540a}, {pid=237, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_73>, region=f5085d50d9508c1b11a3e0ed7e98c007}, {pid=238, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_50>, region=933660db79d8c4c42d9cc864493b7d5b}, {pid=239, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_72>, region=714388e2805cc386d7c48850b29e89a8}, {pid=240, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_16>, region=bef909fc3b5fe93a466007e73f8f79ed}, {pid=241, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_33>, region=f445d91aea86819a2fb4278d5448e178}, {pid=242, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_persistent, region=6d22694b0c619bbdd0b8ffe7cfa2ee73}, {pid=243, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_persistent, region=27729d98bbeaa6169d23642af41bb007}, {pid=244, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.app_flow, region=5e934d2af0695112564acfc62766a171}, {pid=245, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_22>, region=b2e955a6ea739dfe1ad4880b40e12d1d}, {pid=246, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=a9339217627c55dd9e2fcc57f9e70fbd}, {pid=247, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=512394fd78cc3fd00e72796f5e468f82}, {pid=248, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=0b90a930e8da2c15cf3765574a0ac85a}, {pid=249, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=a6fa48fac62e74f6ce14d81ac2a916d8}, {pid=250, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=6783fc26c108ac5d7e4c59c68d050ef7}, {pid=251, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_60>, region=df6752fb754f6a12769c9f8f6868f5fa}, {pid=252, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_42>, region=c7e68ad2b83cf444a3fe55c597abef1c}, {pid=253, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_9>, region=55ca2633ba2ddee3b6efa7e1b477c60d}, {pid=254, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys_locations, region=17ad83a7769cc356132351b13de1d212}, {pid=255, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_61>, region=04761c39e35764683291abfed1bb32d8}, {pid=256, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_22>, region=48750756b3a6e4414384853246bc646d}, {pid=257, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_66>, region=c8be351bcf2f0d0d119c560d1bc46f1c}, {pid=258, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_17>, region=b8f04ac037145bdf82440c6fde20d8da}, {pid=259, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_23>, region=3dbac8b33c33379e1812b12b6a148d32}, {pid=260, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_2>, region=9ca449153035cf901abd8402d29bf318}]
2024-05-06 22:17:11,990 INFO [PEWorker-16] procedure.MasterProcedureScheduler: Took xlock for pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03
2024-05-06 22:17:11,993 INFO [PEWorker-9] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=448, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl, region=52232b406176da8242980a3735e7b551}]
2024-05-06 22:17:11,991 INFO [PEWorker-1] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=261, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_55>, region=9cabdbf8353308c257cf64514c0bfe69}, {pid=262, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_58>, region=fb87ff575b36c5997d8713378e47f89e}, {pid=263, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=d230357193249bbb002f0785f4c7271a}, {pid=264, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=00af8a539d6212c37c174562639d25b0}, {pid=265, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_63>, region=dba7407a60581148a46c4a135932f335}, {pid=266, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:cin, region=4ee39571768335fe74c62626ef424ec5}, {pid=267, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ztw, region=fa12c00dd61aab37437a09f92f9ef8f0}, {pid=268, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_28>, region=d8dabdd9f1b4080679de2e5cf24b53ee}, {pid=269, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=e7eb7c9c11ef47b3b3d77c0aebc6e978}, {pid=270, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=9672799e1d65bc522c12e4ae8443b412}, {pid=271, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=daaa022ec47e538c1bbdaedebbeefdbb}, {pid=272, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_1>, region=b03bd74027c4b06b0dc80451e5864952}, {pid=273, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=f567546ebf7fa1af2c3c1d913a6cec95}, {pid=274, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=ba777b929557fc60d1730984f6faaad4}, {pid=275, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=0692c4f2914cd80dfe661e3c6ba8ab27}, {pid=276, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_68>, region=2119a81b0ae71aec5dfe6a8635454016}, {pid=277, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_56>, region=f37c3c70820e5dd95166c6e0df6c266d}, {pid=278, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_41>, region=de842bc4866c150a1d0e02cf90743bdb}, {pid=279, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_41>, region=6a8808705fb01b06a9719d0263d4e119}, {pid=280, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_28>, region=4efcb1aa3e0f2cda75f0869656b69489}, {pid=281, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_34>, region=22be35f2c3c4d6024565c71803750012}, {pid=282, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=7dc8f2ceaef8ae01d52da98d3381be71}, {pid=283, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_41>, region=c190dcc9ab4138c6bdfcf4d2050954f4}, {pid=284, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_72>, region=4291fd7de0724af0dac6b5390f01b832}, {pid=285, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=eda5d4393ccb8a30964e057cf1585638}, {pid=286, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_28>, region=5a652c833612877e99cd17a18d16e25c}, {pid=287, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_71>, region=af5b7441e4dcc6be7a729a010f640b98}, {pid=288, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=878f434511529251fb80bd0fa615397a}, {pid=289, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=752a7e93a236de68fd172255e8dc59b8}, {pid=290, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=98aa87b3d0b14083769c178732e65521}, {pid=291, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_23>, region=fccde6c1bde39b9f50756232422bc353}, {pid=292, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_3>, region=10ecf35cb2bbe68c17f7bfc232e6ac9e}, {pid=293, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=81aa44df8d1be4ee7aa7b4b21506cb6d}, {pid=294, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=ebb3848ed489a7d12e74033fae54269d}, {pid=295, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_persistent, region=715fc0851a4a80f9570d15f9f42eeaed}, {pid=296, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_55>, region=c3466fc1a892f1cd76f93d31276b55b1}, {pid=297, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_19>, region=b8a6f89dd367215c3661ea2ad312eb2e}, {pid=298, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=25e33c575daa89c42bc891bc3133a438}, {pid=299, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_47>, region=da4740e4384c819e8b4475aafa668d23}, {pid=300, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_21>, region=5acae61fe9dc5ffb5d6d72e4fefe9062}, {pid=301, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_16>, region=16e4f4914e1b1df0ba72f5853fa525d9}, {pid=302, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=auth:account_activity, region=ed108e6facb210154c031fc595d2986f}, {pid=303, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=5fb1730f3bdbac7016cf22b70b5ebb27}, {pid=304, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=f8fca7a15afbc527020fad926c0daead}, {pid=305, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_37>, region=cc19878cdcfadaab0aa41bccffd1bd65}, {pid=306, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_26>, region=0c0437c6e51070e2a1eb57386c3f2bb0}, {pid=307, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_1>, region=7123221617b10eb342325353513d7455}, {pid=308, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_12>, region=86ccef773c3fd228453a37c170aece84}, {pid=309, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt_reference, region=d3534603c564eda83632939a3fa1224c}, {pid=310, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_27>, region=f151a2ac50ca31cfcc64b46f6859fcd5}, {pid=311, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_27>, region=5cfb990ab8940b24cf7f007e7ef0bf7b}, {pid=312, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=5306405ccd189c1ef2d5962da914349d}, {pid=313, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=90ce8e8e81f751303cae2a5772c21795}, {pid=314, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=89a247d379f35e764b0b22504b78f325}, {pid=315, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_48>, region=bdeca0a3f7f9e5b7c9937e6e8a1d1be4}, {pid=316, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=7e3532e48f693d8d289d813148d6a4f5}, {pid=317, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_56>, region=4ab6f8e662514eba09398bcc767febaa}, {pid=318, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_28>, region=a29c8fc117e40eac93e0f9e119553614}, {pid=319, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=3da669b0bdc76c87363281b8e99dfa8d}, {pid=320, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=e57f5bde53018a69c7c0022493fcd354}, {pid=321, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=1192336a35ab7cb2fd8f575cec43fff0}, {pid=322, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_10>, region=278fbd8a05b5d09cb4cb4b409c864584}, {pid=323, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_60>, region=be429de23e05dd59f824f0e12ed4c4d0}, {pid=324, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_25>, region=b7cdce4d0b956eaa69a766672b3090a9}, {pid=325, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=bc8adad0a53341adf39cce1f5bcf3e9d}, {pid=326, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_25>, region=89b61005b4de954cf1bd8a1ed046c294}, {pid=327, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_3>, region=c2890cd6fe28924884b902c60f6ff434}, {pid=328, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_3>, region=d43cb6cb67cb7ae0ddb3a577bfedb707}, {pid=329, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_54>, region=348826fe55734025419e00d13f178218}, {pid=330, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_60>, region=76a52c147a6b738fee00abf0a8d38cfe}, {pid=331, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0de76b556fbaa3bd1a3aa02dcb9c3a3f}, {pid=332, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0238767c59208b1368ec21218f7361ac}, {pid=333, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_49>, region=707c27be94d45223eaeb876b0929793f}, {pid=334, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=bea7ac8a617a7380cd5c2545e420d3ba}, {pid=335, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=c51df52ff432403504fcb9c5b650290c}, {pid=336, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_40>, region=d94808418d2327e2e61a3d9cd141cc5a}, {pid=337, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_11>, region=fe4ce65f39cffa3fa363e07ad51a94b2}, {pid=338, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_5>, region=1a705b0e8eeb0d1f9561ea34a7d2f7f8}, {pid=339, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=4aa2b82b3718192d0c0126855ff9cb05}, {pid=340, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_46>, region=c3e304de319ea07e830463cb8212bcca}, {pid=341, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_57>, region=0a179c22a5ce6f8c93fc3cefe4a1f864}, {pid=342, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_60>, region=0b3ef2bc9d82cc296fe41b46de26ed78}, {pid=343, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_7>, region=0deca40e0bf3442b4c73726b16e19743}, {pid=344, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_12>, region=b541f926df73acfada2e2a35642d7c07}, {pid=345, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_73>, region=aff63924d0be9e25291bdba5d2bcaab1}, {pid=346, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_15>, region=691aab81768ff6a808b605f7532cbd1c}, {pid=347, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_35>, region=8cf9e7c507961c6b6cfd5e09397e274f}, {pid=348, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_34>, region=1e1ab991cd7fc4fcea21491fc4271e50}, {pid=349, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_43>, region=3deadc750ca47a175e7bef12b9396233}, {pid=350, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_55>, region=8bc319cfb1232f5e9d8cdab58fdb73f5}, {pid=351, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_1>, region=ff435f619d4ebca70dccea01c1544419}, {pid=352, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_32>, region=5ee934b10ba0d0a3ad5187e8fc4ebc6a}, {pid=353, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=28e539bfa8ad97ccdd163eccbf272a29}, {pid=354, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_22>, region=44bb488a9e65a184488146ffff903e65}, {pid=355, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=dc9d9c2f14473333d6d05754356d9ac6}, {pid=356, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=1f3a3259f905d22af35549c2db081544}, {pid=357, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_11>, region=c8dc2b7ba47fdd1b075dca81cebdb645}, {pid=358, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_29>, region=510e50899e5fb5e8ccfb71ff63c5638f}, {pid=359, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=989ca9c8a3f5a1756ccdc1089fcc4cc1}, {pid=360, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_6>, region=e8937927d7af6ee8273fdae233beedaf}, {pid=361, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_13>, region=93c56196d1938a5f4136567602317244}, {pid=362, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=8986a928fe5c5e4325198664ce5d17fa}, {pid=363, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:cla, region=8402fe5144954dc56d45adadf67b9cbf}, {pid=364, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=d70ca2c40b9ffc76d58e7a530789a8c0}, {pid=365, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=138cbafeceebde3459edf79fda73d450}, {pid=366, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=cc58fc9fedae3fc75c8655986902ba6c}, {pid=367, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=479d22ee3490e4191ef1d43c17fda93b}, {pid=368, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=5febafd6d035f8e20dcde9a4f74e61ce}, {pid=369, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=b162fa7207b2a21cd616522946bfab3e}, {pid=370, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=4f968e6ede745833d6c639d0a9ff2807}, {pid=371, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=9e4c23ea1ce446781c57a72d59ff75a1}, {pid=372, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=f4e41961f3920e038f0a5aec72a7e667}, {pid=373, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=d382dff4e9eab83e22dfd63943168677}, {pid=374, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:wlt, region=27021a1dd9b3a3cf0ac8d9022746150e}, {pid=375, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zao, region=4f3f5a5bf17be1aa79a03e94c1285576}, {pid=376, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zbl, region=9a948d0c445d3d810f402b9cf77f8a85}, {pid=377, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=bea13c1ef47833992db43210e38ba678}, {pid=378, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_61>, region=816769a8ff9c8d34a484441ef39c2b1a}, {pid=379, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_15>, region=f2d1d8517f4541a4916db74e63e9df0f}, {pid=380, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_70>, region=c6efd6b6836e968552d8f531ff1358c9}, {pid=381, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_25>, region=2e74a82a4b3f6297076ef6e25261b460}, {pid=382, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_7>, region=56666ded0297dd0136797f4844b33a0d}, {pid=383, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=eed5ba2cf6b68aa4ff6149759223c266}, {pid=384, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=e04ad9181d3577dac7c126c66c721650}, {pid=385, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_15>, region=8bc9b9261ecb41b744ae8fcabae8bdfc}, {pid=386, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_65>, region=ee223a2d3c84bd2a35227124ccd3efb8}, {pid=387, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_8>, region=8f388296a24cf746ac8762a16698ca22}, {pid=388, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_58>, region=f86e25e7ca220db5c908669b3de1b1f3}, {pid=389, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_42>, region=05219a664defb595ca7658a1bd7d511a}, {pid=390, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_34>, region=ee0bc298e2fc3b8dd0a15fec2ae894d6}, {pid=391, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_23>, region=96eff57175c090f342530b2f41b2de99}, {pid=392, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_23>, region=c8020dff04475030b1ca6d9fb66475e0}, {pid=393, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_62>, region=9fcc9a48a801657d7abea81a6f41b3b4}, {pid=394, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_9>, region=5e39e64343d157bca285adf8cd1b4753}, {pid=395, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_12>, region=7695f48743f8d50e36b783ba93ce5bd7}, {pid=396, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_53>, region=4265434f477b4eadec5918f00851ea86}, {pid=397, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_72>, region=91c5fa07f5bd2297ea2b1c3837b69407}, {pid=398, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=3f78b16536cf86845f913b6d333d799b}, {pid=399, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.app_flow, region=662005b2e8162740152cdca4d80ab677}, {pid=400, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_14>, region=3b9fe4b67b5a48ca8e5c6ac5fb7006bf}, {pid=401, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_64>, region=ee909bbd56543c5191161088fd5a806c}, {pid=402, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_63>, region=d6d13581484e69ca42f4702a1a6af9f0}, {pid=403, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_63>, region=5d854f1eb7bcda240d0f6d6bb8545815}, {pid=404, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=9d9ccd2bc40a4c358e41a86ee9957c15}, {pid=405, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=00c8319a210d90772fd031a45c88ec36}, {pid=406, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=f3a773f2add03d4e976fa8a5ab200c0c}, {pid=407, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_51>, region=3bf60504f9714870997e24ed142d3faa}, {pid=408, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_72>, region=4b2cca0a0cf2723a8488e88bafacf95e}, {pid=409, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:services, region=07466e00c206b9b934417f0040059a67}, {pid=410, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_22>, region=c3d9ee32a04d228846ea0bee462f4e55}, {pid=411, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_26>, region=abc6e8e483eb33ef88e9026e6ba8dfed}, {pid=412, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=73ef17461282d50cff567f089e432bac}, {pid=413, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=8e92a16af113fd860455d91f37aae3b7}, {pid=414, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_11>, region=16fbec1044017e15b5b3b514b17eb59a}, {pid=415, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_6>, region=6ad83e8d695932a4075f6213fc924e07}, {pid=416, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_24>, region=7ba8496b2e2774e476521870581eb137}, {pid=417, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_72>, region=77afca33126ac4a4d3650471e4f51852}, {pid=418, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_4>, region=4b3368505a01a7db9df97335fa2c24b8}, {pid=419, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_31>, region=c942b0795588e8965cfadeb8b409f510}, {pid=420, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_40>, region=15e502c41f2b932092c32a8613f4948e}, {pid=421, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_23>, region=0d51be0e6f91c2df9e50fb6bd888ae5b}, {pid=422, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_43>, region=e6eb36d4c9a022460a70b71cbb85038f}, {pid=423, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_70>, region=fb283affe3bde8f245524fc2a4465e66}, {pid=447, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_13>, region=3129677490b14ed8ce90f0c386f5af60}]
2024-05-06 22:17:11,991 INFO [PEWorker-7] assignment.AssignProcedure: Starting pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null
2024-05-06 22:17:11,990 INFO [PEWorker-8] assignment.AssignProcedure: Starting pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null
2024-05-06 22:17:12,001 INFO [PEWorker-15] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=449, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr, region=2be9537800624651fdc23200b9d052e6}, {pid=450, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt, region=6d268b7ba4e5a74ddb13a99f210b7aa9}]
2024-05-06 22:17:12,038 INFO [PEWorker-4] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=451, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=c61efe7f834bd5e8fb1ef573c6476027}, {pid=452, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=466adde637ee5d7641d5c2105074b9f4}, {pid=453, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_32>, region=3917850b8d9ad87fe38019f012fe2add}, {pid=454, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_41>, region=eae44441062d5dc8c0688b8f055e60a0}, {pid=455, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_41>, region=46707cfcb32eb52dbe9e8c8203afa443}, {pid=456, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_46>, region=92d3626516835796829a9a45c5956c4b}, {pid=457, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_8>, region=b2bb4b7465cdcfebe64237094ce948d2}, {pid=458, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ztw_locations, region=6adeced16672c2f98bc3c1aca6e55db0}, {pid=459, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:hgg, region=9330c212add67cbecd0e4bbb65f72178}, {pid=460, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys, region=7afb999201aeeea87e994597952c197a}, {pid=461, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_11>, region=1f897c1321e379b023db8dcb955005e0}, {pid=462, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_66>, region=e694b978e0c122633fd837284053f61b}, {pid=463, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_63>, region=30a70ae706e91a222274166de838111a}, {pid=464, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_63>, region=83a74451025580bb7f9d0cd11d10a139}, {pid=465, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=6d3586ba82d879a8041a830e463cee79}, {pid=466, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=1c46655faa39c4ab57339fae984cfbc3}, {pid=467, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_61>, region=53e0297c25320786bb23231acb2b6b04}, {pid=468, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=cacc72e20e62f4067f47519ad1d267ba}, {pid=469, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=a7c7ae317038d7a576618d3fcd27812f}, {pid=470, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_63>, region=b8100430c8dce21a0d29651e1cdfa327}, {pid=471, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_66>, region=a94e776a27155a45f4ae995a595598df}, {pid=472, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_19>, region=f708a1b93798a3a32f57ac88298f44f3}, {pid=473, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_7>, region=8680fb202f5002291d01e33f6ae0c677}, {pid=474, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_64>, region=cb2f8e4706d0671b296cbfa6dd64d80f}, {pid=475, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_39>, region=827abc98319f1cb36091de23bb8d0c4a}, {pid=476, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=e7027e77bc5a3502b93817fe70f5b1a0}, {pid=477, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=f84659ec3a9df6d12f266e23d77dfdbb}, {pid=478, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys_quantifier_index, region=9585cb481dab76c8289ae37a2070e733}, {pid=479, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_2>, region=73ad4e436fe8a4c454ad8cebbb5229dc}, {pid=480, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_3>:<TABLE_5>, region=a5464cc09a771f2bf2eac2d724cc2fea}, {pid=481, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=c1e760cf795e36d5c930114b7a3b6c69}, {pid=482, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=83f8632fc23f1acbf2c1e9766a75b782}, {pid=483, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_66>, region=cc3f02439ba1b94dad139118a39e7fc4}, {pid=484, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=559e67f55cbb06b0130690c943b809ca}, {pid=485, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_30>, region=913134de6319bd6745c9b89c4fc187dd}, {pid=486, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_63>, region=fca99618699b16e3cbecc721c923f547}, {pid=487, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_66>, region=97ec5bc08b8ee3e9b0465f34aaa1b686}, {pid=488, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=e16719230d3f0e3f93ea2812c58b5be1}, {pid=489, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=ce4aba20a0fc64b3a5cb7875bcca9bb0}, {pid=490, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=8b87f38f2951c4ea3c61d07277b7b896}, {pid=491, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_29>, region=174010da7a14cb23ed2d4cda5066f614}, {pid=492, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_1>, region=abbe0ec0457aab7b4153e6f2fd06fa85}, {pid=493, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_39>, region=1116fa6dd513258f3d5affa052962bfa}, {pid=494, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_38>, region=bb186abfb55ebdcc607b27d71668e200}, {pid=495, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_66>, region=e837a6cdeb7ef4c0145ed521200a52e2}, {pid=496, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_67>, region=7c97ccf6990a6eaea378421b8c577e3d}, {pid=497, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_18>, region=4f857594b08360c5522bc4a3d4097997}, {pid=498, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_12>, region=c6bdd90b36bd98a20781219b6af2d303}, {pid=499, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_65>, region=ef6516fb5fe98a69df64db8724b3517b}, {pid=500, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_4>, region=9a8fb840b2caadcb1c179474eaa635b1}, {pid=501, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_42>, region=0a1b3f8f3d99e378328245963da407cc}, {pid=502, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_timespan_index, region=4bbf2884d5af0d06f627ad9d193837d0}, {pid=503, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_26>, region=35c34d62af1e731f278c45961a307836}, {pid=504, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_23>, region=abf3962c01cd35820afc1399f4d6a405}, {pid=505, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_18>, region=ff1798effe6fc9d1f37cedf02e58e798}, {pid=506, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_61>, region=bf08e209b6e07121851d274a628fb856}, {pid=507, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_38>, region=f812525ecdd82c440b8dd8d5e958e2ec}, {pid=508, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_50>, region=e8e3c6220bef865656c4a6d568ca8008}, {pid=509, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=fb8af02c18de3e29eb27f5f1721e4f7a}, {pid=510, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_72>, region=9ee442a3b9ae69e207bcc6411cb72c58}, {pid=511, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_42>, region=480f99176d21b082375a33fe3bf12d89}, {pid=512, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=702cd74f208b5cffd27d45b0177634ba}, {pid=513, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_11>, region=8335322953acc00d697149fa02bc039f}, {pid=514, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_27>:<TABLE_69>, region=66136ae9052fa89bef76c42bef64c009}, {pid=515, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_24>, region=399e943e9954806aaef8d6a2bc70e1fe}, {pid=516, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=59e65b38b13dba3ea251a733ff328e59}, {pid=517, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_59>, region=ca8753bebe73cc6f6d5c664d619c2ae3}, {pid=518, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_27>, region=767032840c65e8f9cd5089a22ee1643b}, {pid=519, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=8d5801804805600bea72ce5cd9446c48}, {pid=520, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_55>, region=bb8faf48d39cb9c5667c4f4b81e37388}, {pid=521, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_45>, region=758702557664af51f95e7892184a84c2}, {pid=522, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=b31f6d9cd79b77f7b387a48f1578f283}, {pid=523, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=85f527a85b1459ba4d0515cba9f33d4f}, {pid=524, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_49>, region=a501efad6e61a886accd69096a51de0f}, {pid=525, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_55>, region=d685720514e4b88309023b6a502966b0}, {pid=526, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_43>, region=49d9e7be19851aa294cc7cc4b55d9e40}, {pid=527, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zgb_persistent, region=8f72a863e1c3d5720cd3bbe6c7971976}, {pid=528, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_25>, region=25b787ba9ac7be4bfaf16d4d5a48b1c7}, {pid=529, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_72>, region=12cee6e8df27cb239ff357e0cce911f1}, {pid=530, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_26>, region=3667a7af201512c963c36928d0beec2e}, {pid=531, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_9>, region=a4ef2c4778117fd18df53fbe2db96e16}, {pid=532, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_27>, region=3e9a743da8b6729ebcf46c11dd9592c8}, {pid=533, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_42>, region=c0dda7faac9764443d415ace31bd457f}, {pid=534, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_6>, region=658d35c14ffcf80187acf4a7e588d8b5}, {pid=535, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_20>, region=3f8100c740920f820672a13942110ced}, {pid=536, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_53>, region=a8886ffa4fe8011cf3644aaa8686f5cf}, {pid=537, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_60>, region=79c7205c1a713ccb93057d36df90fa4a}, {pid=538, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=588b3fe66bd63bc7b9aec55554e6baf8}, {pid=539, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=5dd7d57f8858f77f59493d767577fbe6}, {pid=540, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=85e30b21a4d068e598b850cf405b5a09}, {pid=541, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=8d02e4646ea1e3b269b0df8667c49b5b}, {pid=542, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=24636244d51e67e3a35843d2b44b0e06}, {pid=543, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=2438f52750eed160df63520c76e04bc7}, {pid=544, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_19>, region=09a5df83079bf8b9e2f0ce3e05d35e9e}, {pid=545, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_27>, region=d3d4f6e36922bf18cf888da9e48daa8a}, {pid=546, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_6>, region=4e32e01434dfe469101bb291c037b861}, {pid=547, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_53>, region=9d63646dcf52a57e26314aafda41ac3b}, {pid=548, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_37>, region=ab3395aef65ed976125fe46e5a837d36}, {pid=549, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_71>, region=c32c2e3905359e69ca142395a088cf97}, {pid=550, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_11>, region=3bfbdc17c6f863c691e9795dacb68c4e}, {pid=551, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_34>, region=c1860525b46b5bd2f9586d2c87504820}, {pid=552, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=4358f469f4d74d50d66331ee74277e02}, {pid=553, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_43>, region=9ae4e344d28cf89d9781d9b3cbb2e73b}, {pid=554, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=0e0051bd2e1b2cfa869a623cdc909763}, {pid=555, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=c6b220e50e28a3b3bf4a5f29fab3f2cd}, {pid=556, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=92dfdd8a2532cb399e6e67eba5664a79}, {pid=557, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=0038e7ee32d606dc00bace075e151fc5}, {pid=558, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_24>, region=02bb3db34a8e992383401b41879a8ad6}, {pid=559, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=3e19e390336d24abb392ce5d53081cd1}, {pid=560, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=752425315bec32b835fd86bcc9e1d955}, {pid=561, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:cla, region=0e8e45dc138f0aafc891a7611c2b0cb7}, {pid=562, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=376d53328d29de238443148140de46a8}, {pid=563, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=b4c74b65072baf60835a410145b819c1}, {pid=564, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=d1194857a1fa4d9c9c2d57a2e190d5ff}, {pid=565, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=47b0efbd946486e7bc824ec5a9150aac}, {pid=566, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=c85e4a00d06036294f35f263269bda4f}, {pid=567, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=e6c0e6d7bbe6ad01be65e9fe57aba426}, {pid=568, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=cd8e266bde4dc549910a88146e070553}, {pid=569, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:hgg, region=89b70ce278744658a639eddf6d90f44d}, {pid=570, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=89ed47dfd54aaacce7298ed7ffa135dd}, {pid=571, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=510c369b6cf0c330a9415854efd36873}, {pid=572, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=4a49c9bac8263915c77b8b116f85e5f2}, {pid=573, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=f315c4987d7f8e96d35b5a5b836c18a0}, {pid=574, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=36546490b285ad7f21af0b19007480f7}, {pid=575, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=00cfea50f773a27134e67aac6dce8fa1}, {pid=576, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=0c4ffaf5aeca347a19cf9eed3ddc5358}, {pid=577, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=509d458e718f97e075e26d10be568e3b}, {pid=578, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=09db0b1c02f96a50d1fff02f1c8af7db}, {pid=579, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5e800770355a88f427488b629abb57cd}, {pid=580, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=83f7dc30a5f06d5adb13524588855818}, {pid=581, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=291b502a62b078b5586d649885f5d475}, {pid=582, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=db1e76f15e3d73523ebdaf1c1c06abb2}, {pid=583, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zao, region=38e95d30bba9428234fb15221dc6ce5f}, {pid=584, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_12>, region=9b1f61309b94ada8009a178ddb4e4636}, {pid=585, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_59>, region=8641d6490c9b56c408d94ac096740e3d}, {pid=586, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_46>, region=354f0c8a1370815286b51a4b1582d3f8}, {pid=587, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_14>, region=5b2dcd3b95616450baf72ee664188293}, {pid=588, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_29>, region=6d9ca2a6acfe4ac471cbccdc120c3b04}, {pid=589, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_24>, region=2e2b88b0bfa20453c49028eced20c2ee}, {pid=590, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_7>, region=6a8efda89913a2b556771c59c3047345}, {pid=591, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_27>, region=24c5ee2022b934078c0f78d21deff906}, {pid=592, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_reference, region=dc0291681b786728f6cd21ee6a23bb6f}, {pid=593, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_42>, region=342672c1865f212a3f0fe41dfe989b86}, {pid=594, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_61>, region=a56d1d3443e867c6f2e4d4204556589c}, {pid=595, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_54>, region=dbac96812aba03d24e549e0b89c8650d}, {pid=596, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_35>, region=3d41647bd0a155cfed283088a1179728}, {pid=597, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_34>, region=3415b85569434236da100aaae8fd01ca}, {pid=598, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_30>, region=d8582c530ca270bea5c3fb3bef040de3}, {pid=599, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_25>, region=2490192492c7652d71ddd69df4f8b8c1}, {pid=600, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=5d4fc7dc0f39106a648bcf47f1552680}, {pid=601, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=5d76b5dcd749c89e2ea2977711a06c7d}, {pid=602, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=6595469e6bb9dfe0e58100daad516e03}, {pid=603, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_31>, region=9ba2f6b08e1da4bd2d47a4df7a7032aa}, {pid=604, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_57>, region=8fcc91a518f6f34a036b6c5e6ee7f4c9}, {pid=605, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_34>, region=bf077734bf83bf98d2d1f078fa2dfcf5}, {pid=606, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_26>, region=d51433bc454a247671a6fc8d8cc34507}, {pid=607, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_61>, region=b8b53b12a6855eaaf23f2fcd86d80935}, {pid=608, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_65>, region=e6955787e6997e9f09d6e685031955ef}, {pid=609, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_71>, region=5560528e9a2a93d789b7be1fc22a2c16}, {pid=610, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_72>, region=fc009e8566c9b8db9fef4977d10f6cb5}, {pid=611, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_29>, region=788e6997db951c8ea7fb9d637f8487d8}, {pid=612, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_25>, region=ee67ec547647c28fdeed0101b8296681}, {pid=613, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=3e0baf03cc1879ce98f594480ffe4632}, {pid=614, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_59>, region=0f168557d185be24d8236b1f577e7c7b}]
2024-05-06 22:17:12,053 INFO [PEWorker-5] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=669, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=75a9ecc30a2fd3276de9ac01c4a3b6af}, {pid=670, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=7e82b7202431c0772d57b96c859c815d}, {pid=671, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao, region=585a71127b4393c583f2b1a39536009e}, {pid=672, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=b077d9c6af116f153cfbbf156dde77f6}, {pid=673, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_57>, region=ed80d78492d5b36bd676daf7dff5aa26}, {pid=674, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=fc8f9b834ef15ada068f9054f0a3246e}, {pid=675, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=e14adc8133fa8e50791a7ea74c88b1ee}, {pid=676, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_60>, region=55c76fd6461d42a68ec583d1be33b43a}, {pid=677, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_53>, region=07baa87f9c9c904c72163ec6e2d9adb3}, {pid=678, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_43>, region=15de5b3bdebed284129ddaf35009b932}, {pid=679, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=ffe92559774b50614e7551275f6594dd}, {pid=680, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_67>, region=1b814c1cbf8c1e9de7565efd3b25f990}, {pid=681, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt_timespan_index, region=0adb337e4d98c90d5301d9964f4282ea}, {pid=682, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_5>, region=c9826b585da29b6840208c1662c03086}, {pid=683, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=d8a1bea07427eb558c39190f8f946746}, {pid=684, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_46>, region=7def6eddc52abcd19bab85839914a1a1}, {pid=685, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_71>, region=dc099e9ba7d0d3f5f9aa6be10b80028e}, {pid=686, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=8db2e5cada5820754b004acbc880ce17}, {pid=687, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=c799db86045094fd0320e5fb37323173}, {pid=688, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_14>, region=0bca31436f8ae42d4d06ef1dffb6b7fe}, {pid=689, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_by_query, region=9c7f4d8ef637f8e2fb0ead64bdcc5c36}, {pid=690, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_by_query, region=fd6b15de65bba3d43436505017e5b2ea}, {pid=691, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=713e83fb0544e3f507c4cfe1b9f510bf}, {pid=692, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0e79e0171eb7ed47389b4eb984fca553}, {pid=693, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=09ed5c26faffb87ca8c6843a0a7323ed}, {pid=694, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_8>, region=8e267c07f543e25bc2a320707a204d21}, {pid=695, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_10>, region=a362ff12ae734f09c9dad5e2cff7dddb}, {pid=696, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=77127d1dca307dc74ccae9e12a23078e}, {pid=697, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=8b4402597026de84717ff39705f490b5}, {pid=698, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=43c366fdcd5094a13209c7bae30b7a9c}, {pid=699, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=9469573427d2c76940adac29705267ae}, {pid=700, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=55617f37c3820f05aad9aef85e4d9346}, {pid=701, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=16e0ce258462e89603eb7697228ab6f7}, {pid=702, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=ecea80ddf358a89c803ee0f1a8cc8b5f}, {pid=703, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=1c4091e7e7cedafb58beea23cd2959c2}, {pid=704, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=50f17b259f03ce2f7bf7ebe0b41c5e45}, {pid=705, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=1d3569bdb8ae48721d91d0783e77cd92}, {pid=706, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_48>, region=b2a4ab25b11f009e6607d70bc9ffcf3e}, {pid=707, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_46>, region=dbdbc86496bb9c98e638323bbefeee5c}, {pid=708, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_56>, region=d41f6b1862d2ae11100d85565f8e5fec}, {pid=709, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_37>, region=7436bb293ecaffb269b22bf9af7c8576}, {pid=710, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_client_timeline, region=d8eaadd1083d3a03848396f2c8d8c8ef}, {pid=711, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=2fbd1ff98a1b2442e718c43ac48f4734}, {pid=712, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_66>, region=e06908bb602a88955af02e853dbe37df}, {pid=713, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_3>:<TABLE_31>, region=177abc45acf52c795b6dd6f67145c5d6}, {pid=714, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=9fa16bd1cdcd629bccd2f6ffd381a7e5}, {pid=715, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_28>, region=e94e6814d161dffd12834ae227ffef45}, {pid=716, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_63>, region=ff9ceda3a8b8bd17fc4d627476ba2255}, {pid=717, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_1>, region=301e73eb5950ae0b8b5ba2b97a30ca13}, {pid=718, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_53>, region=099257cfe303f0c20275b4a8e8c3e276}, {pid=719, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_62>, region=8172981d31c57437920ab13f6bea9a4c}, {pid=720, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_57>, region=59425dfd7861755acfcaf4a741a9e402}, {pid=721, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_26>, region=8d9d3eccb0b62fb9746166f35139169d}, {pid=722, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:demo, region=370539314e0bf6e2133520805e16e594}, {pid=723, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_41>, region=fca59829f9914d8bc0ef56065f18386e}, {pid=724, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_51>, region=00e62790dc24813dce28875ce8cd5b3f}, {pid=725, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_40>, region=0d9ed0f5f7fc63838db85dc1c406c54e}]
2024-05-06 22:17:12,097 INFO [PEWorker-11] procedure.MasterProcedureScheduler: Took xlock for pid=19, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=ba37bf35a4430884d1a6ebbca95a8387
2024-05-06 22:17:12,097 INFO [PEWorker-16] assignment.AssignProcedure: Starting pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null
2024-05-06 22:17:12,099 INFO [PEWorker-14] procedure.MasterProcedureScheduler: Took xlock for pid=20, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=68531e269838537d534e81683e2a7097
2024-05-06 22:17:12,100 INFO [PEWorker-6] procedure.MasterProcedureScheduler: Took xlock for pid=21, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_28>, region=72f61b803e206cc95dc2a05b1112d39f
2024-05-06 22:17:12,100 INFO [PEWorker-10] procedure.MasterProcedureScheduler: Took xlock for pid=22, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_5>, region=f696c18f229b7b6525405d55afdd57cb

Skipping INFO but keeping WARNs here. We have seen these before

2024-05-06 22:17:20,363 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_1>:<TABLE_51>, region=ec4e4dfe2c2bcf372ab8185150d41e1c reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_4>:<TABLE_48>, region=7e3c25195bd235046fa37428a5a58eb7 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_5>:<TABLE_3>, region=ed1abdd5978b355bc547f3322bbc79bd reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_7>:<TABLE_28>, region=319217e06a1140bbf8d3e569e1931db3 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
...

And more of them:

2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_4>:<TABLE_33>, region=d41e7923f187ee13d081591227c7805c reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running
2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_5>:<TABLE_40>, region=7cc3121f8462d670c938bd3844861436 reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running
2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_8>:<TABLE_12>, region=2788b213d23851dc98611dfc43935e9e reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running
2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_9>:<TABLE_68>, region=460607d7a84d9b03c58f853ecb5a2b73 reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running
...

And much more of these:

2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_4>:<TABLE_48>, region=7e3c25195bd235046fa37428a5a58eb7 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_5>:<TABLE_3>, region=ed1abdd5978b355bc547f3322bbc79bd reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_7>:<TABLE_28>, region=319217e06a1140bbf8d3e569e1931db3 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_7>:<TABLE_52>, region=a989864b76285718567c090494024e34 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
...

Re: Tables regios stuck in RIT

MrNicen — Fri, 10 May 2024 07:16:18 GMT

We tried decommissioning both busiest region servers, which had combined 80% of all regions. Decommsisioning fails because of the same Stuck In Transition. Luckily the side effect of decommissioning is all other regions did move to the remaining 12 servers, which makes that these two servers now have combined 3% of all regions. To our customers we have a much more balanced and therefor snappy cluster now. But as long as we have these Stuck In Transition, autobalancing will not happen, and at some point will have a imbalance again.

It seems that the cause is that these 134 regions remain in OPENING, and should become OPEN, which is not going to happen. What is the best approach to mitigate this? Should we use hbck2 to set the region state manually to OPEN?

Re: Tables regios stuck in RIT

rki_ — Fri, 10 May 2024 14:09:18 GMT

If the Region server reports that the region is already OPEN, try to scan the hbase:meta table from the hbase shell and check what state that region is in.

If its still in OPENING state in meta, try to change its state to OPEN. Do it for one of the region and see if that brings down the RIT count to 133.

Re: Tables regios stuck in RIT

MrNicen — Wed, 15 May 2024 09:23:47 GMT

The current stat in hbase:meta is OPEN not OPENING. I can try setting it with HBCK2 to OPEN but i don't think this will change anything.

Re: Tables regios stuck in RIT

rki_ — Wed, 15 May 2024 12:29:21 GMT

If the current stat in hbase:meta is OPEN, I won't suggest performing any other action to change the state in meta. I suspect there are some procedures running in the backend trying to assign the region again. Do you see any procedure as such in the "Procedure & locks" section under the Hbase Master Web UI?

Re: Tables regios stuck in RIT

MrNicen — Thu, 16 May 2024 07:21:55 GMT

There are currently no locks/procedures.

Re: Tables regios stuck in RIT

MrNicen — Thu, 19 Dec 2024 08:25:44 GMT

We still have 99 RIT's, how can we delete tables that are stuck?