Skip to main content
MrNicen
MrNicen
Silver ⭐️⭐️⭐️
May 5, 2024
Question

Tables regios stuck in RIT

2024-05-04 15:27:08,590 WARN [RpcServer.default.FPBQ.Fifo.handler=26,queue=1,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<SERVER>,16020,1714828981071, table=<NAMEAPACE>:<TABLE>, region=275bbfd7b7044afcba2deeeebf622b07 reported OPEN on server=l230-n2.<SERVER>,16020,1714828981071 but state has otherwise AND NO procedure is running

We have about 300 RIT and there alle stuck under the procedures & locks (<hbase master>:16010/procedures.jsp) we have some in state waiting_timeout.

We all ready tried restarting the cluster (hbase/zk) but got only whorse. There are 14 nodes in the cluster 3 are not assigned and we have a couple (about 4) that do all the work.

Where still on hbase 2.0.2  so we can't use hbck2 (2.0.3 >) and in hbase 2.0.2 hbck repair is disabled. Our average region count per server is about 330.

1. how do i get my regions unstuck?

 

    15 replies

    VidyaSargur
    Community Manager
    VidyaSargur
    Community Manager
    May 6, 2024

    @MrNicen, Welcome to our community! To help you get the best possible answer, I have tagged in our HBase experts @rki_ @smdas @SVB  who may be able to assist you further.

    Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

    Translate
    Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
    rki_
    Gold ⭐️
    rki_
    Gold ⭐️
    May 6, 2024

    @MrNicen firstly, You will need to check the RS log of l230-n2.<SERVER>. As per the WARn, the region is already reported as OPEN by the same RS on which its trying again to open and going into a OPENING state.

    It seems there are SCPs/ multiple assign procedures are running in the backend trying to open a region that is already open. 

    Considering that you are on a version that can neither use hbck2 jar nor the hbck1 commands,  you can give try the below :

    1) Stop both the Master

    2) Move the contents of MasterProcWALs to a backup location

    # hdfs dfs -mv /hbase/MasterProcWALs/* /tmp/

    3) Start the Masters.

    If the above doesn't solve the issue, I suggest raising a support case with Cloudera to review the logs.

    Translate
      MrNicen
      MrNicenAuthor
      Silver ⭐️⭐️⭐️
      May 7, 2024

      We tried the solution and the number of RIT went down to about 134. but still not zero. Most of the RS log's are "clean". In de hbase master log we get a nasty.

      2024-05-06 22:34:22,745 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=l230-n2.SERVER,16020,1715026584782, table=<NAMESPACE>:<TABLE>, region=e1b0976c6c5d759............

      and 

      2024-05-06 22:26:25,230 WARN [RpcServer.default.FPBQ.Fifo.handler=33,queue=3,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.SERVER,16020,1715026585040, table=<NAMESPACE>:<TABLE>, region=d41e7923f187ee13d081591227c7805c reported OPEN on server=l225-n2.SERVER,16020,1715026585040 but state has otherwise AND NO procedure is running

      these lines are repeated ever x sec's and are sets off 134 lines i guess the same as the number of stuck regions

      I found out a can use hbck2 but have to add the "--skip version check option"


      *Note had some trouble with my accounts, so resetted the wrong one.

      Translate
        rki_
        Gold ⭐️
        rki_
        Gold ⭐️
        May 7, 2024

        Try to bypass the stuck procedures using the hbck2 Jar and take the Region servers for a restart.

        Translate
        Mr-nicen
        Mr-nicen
        Silver ⭐️
        May 6, 2024

        Sorry wrong user.

        Translate
          MrNicen
          MrNicenAuthor
          Silver ⭐️⭐️⭐️
          May 7, 2024

          Together with some colleagues, we have analysed the logs after restarting HBase, and made the following lay mans analysis of the last startup log:

          These seem innocent. HMaster starts en the RSs are not online yet:

          2024-05-06 22:16:38,187 INFO [Thread-16] replication.ReplicationPeersZKImpl: Added peer cluster=l200-n4.<DOMAIN_M>,l201-n4.<DOMAIN_M>,l204-n4.<DOMAIN_M>:2181:/hbase-unsecure
          2024-05-06 22:16:38,201 WARN [Thread-16] master.ServerManager: Server l224-n2.<DOMAIN_G>,16020,1659400236292 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n2.<DOMAIN_G>,16020,1659400236292]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l227-n2.<DOMAIN_G>,16020,1651336158836 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l227-n2.<DOMAIN_G>,16020,1651336158836]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n2.<DOMAIN_G>,16020,1713307724458 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n2.<DOMAIN_G>,16020,1713307724458]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l225-n1.<DOMAIN_G>,16020,1651396933254 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l225-n1.<DOMAIN_G>,16020,1651396933254]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l230-n2.<DOMAIN_G>,16020,1651396881056 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l230-n2.<DOMAIN_G>,16020,1651396881056]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n1.<DOMAIN_G>,16020,1660153454095 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n1.<DOMAIN_G>,16020,1660153454095]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l226-n2.<DOMAIN_G>,16020,1651397141435 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l226-n2.<DOMAIN_G>,16020,1651397141435]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l229-n1.<DOMAIN_G>,16020,1651336158866 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n1.<DOMAIN_G>,16020,1651336158866]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l228-n2.<DOMAIN_G>,16020,1651396801801 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l228-n2.<DOMAIN_G>,16020,1651396801801]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l227-n1.<DOMAIN_G>,16020,1651336158636 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l227-n1.<DOMAIN_G>,16020,1651336158636]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n1.<DOMAIN_G>,16020,1651336158334 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n1.<DOMAIN_G>,16020,1651336158334]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l226-n1.<DOMAIN_G>,16020,1651336158228 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l226-n1.<DOMAIN_G>,16020,1651336158228]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l229-n2.<DOMAIN_G>,16020,1651397094435 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n2.<DOMAIN_G>,16020,1651397094435]
          2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l228-n1.<DOMAIN_G>,16020,1651336158675 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l228-n1.<DOMAIN_G>,16020,1651336158675]
          2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l230-n1.<DOMAIN_G>,16020,1651336158117 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l230-n1.<DOMAIN_G>,16020,1651336158117]
          2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l225-n2.<DOMAIN_G>,16020,1651336158291 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l225-n2.<DOMAIN_G>,16020,1651336158291]
          2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l229-n1.<DOMAIN_G>,16020,1713308077605 is not currently online. Ignoring request to add it to draining list.
          2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n1.<DOMAIN_G>,16020,1713308077605]


          Here HMaster node l226-n3 becomes Active/primary and the SCPs are being started:


          2024-05-06 22:16:38,203 INFO [Thread-16] master.HMaster: Active/primary master=l226-n3.<DOMAIN_G>,16000,1715026581275, sessionid=0x28f43c489256548, setting cluster-up flag (Was=true)
          2024-05-06 22:16:38,282 INFO [Thread-16] master.HMaster: hbase:meta {1588230740 state=OPEN, ts=1715026596449, server=l225-n2.<DOMAIN_G>,16020,1714828981130}
          2024-05-06 22:16:38,302 INFO [Thread-16] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1715026628302
          2024-05-06 22:16:38,306 INFO [Thread-16] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=l228-n3.<DOMAIN_G>:2181,l229-n3.<DOMAIN_G>:2181,l230-n3.<DOMAIN_G>:2181
          2024-05-06 22:16:38,306 INFO [Thread-16] zookeeper.ZooKeeper: Initiating client connection, connectString=l228-n3.<DOMAIN_G>:2181,l229-n3.<DOMAIN_G>:2181,l230-n3.<DOMAIN_G>:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@4e09d8ce
          2024-05-06 22:16:38,316 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Opening socket connection to server l230-n3.<DOMAIN_G>/10.226.230.13:2181. Will not attempt to authenticate using SASL (unknown error)
          2024-05-06 22:16:38,320 INFO [PEWorker-1] procedure.ServerCrashProcedure: Start pid=8, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l225-n2.<DOMAIN_G>,16020,1714828981130, splitWal=true, meta=true
          2024-05-06 22:16:38,321 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.226.226.13:40378, server: l230-n3.<DOMAIN_G>/10.226.230.13:2181
          2024-05-06 22:16:38,320 INFO [PEWorker-8] procedure.ServerCrashProcedure: Start pid=7, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l228-n1.<DOMAIN_G>,16020,1714828981415, splitWal=true, meta=false
          2024-05-06 22:16:38,320 INFO [PEWorker-10] procedure.ServerCrashProcedure: Start pid=10, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l225-n1.<DOMAIN_G>,16020,1714829249747, splitWal=true, meta=false
          2024-05-06 22:16:38,320 INFO [PEWorker-7] procedure.ServerCrashProcedure: Start pid=6, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l229-n2.<DOMAIN_G>,16020,1714828981440, splitWal=true, meta=false
          2024-05-06 22:16:38,320 INFO [PEWorker-9] procedure.ServerCrashProcedure: Start pid=9, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l227-n2.<DOMAIN_G>,16020,1714829249989, splitWal=true, meta=false
          2024-05-06 22:16:38,320 INFO [PEWorker-6] procedure.ServerCrashProcedure: Start pid=5, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l224-n1.<DOMAIN_G>,16020,1714828981244, splitWal=true, meta=false
          2024-05-06 22:16:38,320 INFO [PEWorker-4] procedure.ServerCrashProcedure: Start pid=3, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l228-n2.<DOMAIN_G>,16020,1714828981422, splitWal=true, meta=false
          2024-05-06 22:16:38,320 INFO [PEWorker-5] procedure.ServerCrashProcedure: Start pid=4, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l226-n2.<DOMAIN_G>,16020,1714828981235, splitWal=true, meta=false
          2024-05-06 22:16:38,320 INFO [PEWorker-2] procedure.ServerCrashProcedure: Start pid=1, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l224-n2.<DOMAIN_G>,16020,1714828981192, splitWal=true, meta=false
          2024-05-06 22:16:38,321 INFO [PEWorker-12] procedure.ServerCrashProcedure: Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l230-n2.<DOMAIN_G>,16020,1714828981071, splitWal=true, meta=false
          2024-05-06 22:16:38,321 INFO [PEWorker-13] procedure.ServerCrashProcedure: Start pid=13, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l230-n1.<DOMAIN_G>,16020,1714828981173, splitWal=true, meta=false
          2024-05-06 22:16:38,321 INFO [PEWorker-11] procedure.ServerCrashProcedure: Start pid=11, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l226-n1.<DOMAIN_G>,16020,1714829249771, splitWal=true, meta=false
          2024-05-06 22:16:38,321 INFO [PEWorker-14] procedure.ServerCrashProcedure: Start pid=14, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l227-n1.<DOMAIN_G>,16020,1714828981420, splitWal=true, meta=false
          2024-05-06 22:16:38,321 INFO [PEWorker-3] procedure.ServerCrashProcedure: Start pid=2, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l229-n1.<DOMAIN_G>,16020,1714828981427, splitWal=true, meta=false


          Zookeeper connection:

          2024-05-06 22:16:38,326 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Session establishment complete on server l230-n3.<DOMAIN_G>/10.226.230.13:2181, sessionid = 0x38f43c48945656c, negotiated timeout = 60000


          House keeping of WALs in Thread-16

          2024-05-06 22:16:38,327 INFO [Thread-16] cleaner.CleanerChore: Cleaner pool size is 1
          2024-05-06 22:16:38,328 INFO [Thread-16] cleaner.LogCleaner: Creating OldWALs cleaners with size=2


          In PEWorker-9 more WAL management specifically related to l225-n2 (Why?):

          2024-05-06 22:16:38,362 INFO [PEWorker-9] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting is empty dir, no logs to split
          2024-05-06 22:16:38,362 INFO [PEWorker-9] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting] for [l225-n2.<DOMAIN_G>,16020,1714828981130]
          2024-05-06 22:16:38,365 INFO [PEWorker-9] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting] in 3ms


          In PEWorker-9 initialisation of the AssignProcedure

          2024-05-06 22:16:38,397 INFO [PEWorker-9] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}]


          Zookeeper Replication Peers, since we have replication enabled

          2024-05-06 22:16:38,403 INFO [Thread-16] replication.ReplicationPeersZKImpl: Added peer cluster=l200-n4.<DOMAIN_M>,l201-n4.<DOMAIN_M>,l204-n4.<DOMAIN_M>:2181:/hbase-unsecure


          Still waiting for the first RS to call in:

          2024-05-06 22:16:38,404 INFO [Thread-16] master.HMaster: Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1715026584479, completionTime=-1
          2024-05-06 22:16:38,404 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=0; waited=0ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms

           

          Translate
          MrNicen
          MrNicenAuthor
          Silver ⭐️⭐️⭐️
          May 7, 2024

          Somehow it was decided that for table hbase:meta the region should enter state RUNNABLE on RS l225-n2.
          Why l225-n2? Note, rit is still reported offline.

          2024-05-06 22:16:38,408 INFO [PEWorker-14] procedure.MasterProcedureScheduler: Took xlock for pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740
          2024-05-06 22:16:38,417 INFO [PEWorker-14] assignment.AssignProcedure: Starting pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=l225-n2.<DOMAIN_G>,16020,1714828981130; forceNewPlan=false, retain=true target svr=null

          Here l227-n2 is calling in and Thread-16 reacts.

          2024-05-06 22:16:38,536 INFO [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l227-n2.<DOMAIN_G>,16020,1715026585628
          2024-05-06 22:16:38,555 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=1; waited=151ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms


          The following may be the first relevant WARN. Its is about ServerStateNode for l227-n2 being empty. Are these nodes persisted in Zookeeper?

          2024-05-06 22:16:38,570 WARN [PEWorker-10] assignment.AssignmentManager: Why is ServerStateNode for l227-n2.<DOMAIN_G>,16020,1715026585628 empty at this point? Creating...
          2024-05-06 22:16:38,570 INFO [PEWorker-10] assignment.AssignProcedure: Early suspend! pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=l227-n2.<DOMAIN_G>,16020,1715026585628


          Next, all RSs are registering themselves and ephemeral node's are being created (I think in in ZK):

          2024-05-06 22:16:38,579 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l227-n2.<DOMAIN_G>,16020,1715026585628]
          2024-05-06 22:16:39,409 INFO [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l228-n1.<DOMAIN_G>,16020,1715026585682
          2024-05-06 22:16:39,432 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l228-n1.<DOMAIN_G>,16020,1715026585682]
          2024-05-06 22:16:39,457 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=2; waited=1053ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
          2024-05-06 22:16:40,100 INFO [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l224-n2.<DOMAIN_G>,16020,1715026584653
          2024-05-06 22:16:40,108 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=3; waited=1704ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
          2024-05-06 22:16:40,112 INFO [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l230-n2.<DOMAIN_G>,16020,1715026584782
          2024-05-06 22:16:40,115 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l226-n1.<DOMAIN_G>,16020,1715026584512
          2024-05-06 22:16:40,116 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l226-n2.<DOMAIN_G>,16020,1715026584780
          2024-05-06 22:16:40,120 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l230-n1.<DOMAIN_G>,16020,1715026584638
          2024-05-06 22:16:40,132 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l224-n2.<DOMAIN_G>,16020,1715026584653]
          2024-05-06 22:16:40,135 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l230-n2.<DOMAIN_G>,16020,1715026584782]
          2024-05-06 22:16:40,135 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l226-n1.<DOMAIN_G>,16020,1715026584512]
          2024-05-06 22:16:40,135 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l226-n2.<DOMAIN_G>,16020,1715026584780]
          2024-05-06 22:16:40,142 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l230-n1.<DOMAIN_G>,16020,1715026584638]
          2024-05-06 22:16:40,154 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l225-n1.<DOMAIN_G>,16020,1715026584877
          2024-05-06 22:16:40,158 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=8; waited=1754ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
          2024-05-06 22:16:40,175 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l225-n1.<DOMAIN_G>,16020,1715026584877]
          2024-05-06 22:16:40,323 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l224-n1.<DOMAIN_G>,16020,1715026584891
          2024-05-06 22:16:40,343 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l224-n1.<DOMAIN_G>,16020,1715026584891]
          2024-05-06 22:16:40,359 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=9; waited=1955ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
          2024-05-06 22:16:40,528 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l225-n2.<DOMAIN_G>,16020,1715026585040
          2024-05-06 22:16:40,529 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l229-n1.<DOMAIN_G>,16020,1715026584835
          2024-05-06 22:16:40,531 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l229-n2.<DOMAIN_G>,16020,1715026584960
          2024-05-06 22:16:40,549 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l225-n2.<DOMAIN_G>,16020,1715026585040]
          2024-05-06 22:16:40,551 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l229-n1.<DOMAIN_G>,16020,1715026584835]
          2024-05-06 22:16:40,552 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l229-n2.<DOMAIN_G>,16020,1715026584960]
          2024-05-06 22:16:40,559 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=12; waited=2155ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
          2024-05-06 22:16:40,840 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l227-n1.<DOMAIN_G>,16020,1715026585107
          2024-05-06 22:16:40,861 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=13; waited=2457ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
          2024-05-06 22:16:40,864 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l227-n1.<DOMAIN_G>,16020,1715026585107]
          2024-05-06 22:16:41,088 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l228-n2.<DOMAIN_G>,16020,1715026585349
          2024-05-06 22:16:41,112 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=2707ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms
          2024-05-06 22:16:41,112 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l228-n2.<DOMAIN_G>,16020,1715026585349]


          Hold on, we are almost reaching the point where the **bleep** hits the fan. The region of htable hbase:meta is dispatched to l227-n2, no mention anymore of rit=offline. But note that the RS has changed from l225-n2 to l227-n2, replicaid=0 in Zookeeper

          2024-05-06 22:16:42,579 INFO [PEWorker-7] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2.<DOMAIN_G>,16020,1715026585628
          2024-05-06 22:16:42,585 INFO [PEWorker-7] assignment.RegionTransitionProcedure: Dispatch pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740


          Server manager is awaiting even more RSs, but we have 14 and they are all registered at this point.

          2024-05-06 22:16:42,615 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=4211ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-1504ms

          Now from PEWorker-8 Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2 and finally the ProcedureExecutor reports state=SUCCESS for AssignProcedure table=hbase:meta:
           

          2024-05-06 22:16:43,572 INFO [PEWorker-8] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2.<DOMAIN_G>,16020,1715026585628
          2024-05-06 22:16:43,583 INFO [PEWorker-8] procedure2.ProcedureExecutor: Finished subprocedure pid=15, resume processing parent pid=8, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=l225-n2.<DOMAIN_G>,16020,1714828981130, splitWal=true, meta=true
          2024-05-06 22:16:43,584 INFO [PEWorker-8] procedure2.ProcedureExecutor: Finished pid=15, ppid=8, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 5.1780sec, unfinishedSiblingCount=0

           
          Time ticks on and no more region server registrations:
           

          2024-05-06 22:16:44,119 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=5714ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-3007ms
          2024-05-06 22:16:45,623 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=7219ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-4512ms

           
           
          No we arrive at the point where the problem manifests itself:
           

          2024-05-06 22:16:45,644 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740
          2024-05-06 22:16:45,644 WARN [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628

           
          From her on I am skipping the INFO logs of the ServerManager Waiting on regionserver count=14. More WARNs:


          2024-05-06 22:16:48,650 WARN [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628
          2024-05-06 22:16:51,655 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740
          ...

          Several similar errors

           


          Than we arrive at master is running:
           

          2024-05-06 22:17:08,175 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=29771ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-27064ms
          2024-05-06 22:17:08,426 INFO [Thread-16] master.ServerManager: Finished waiting on RegionServer count=14; waited=30021ms, expected min=1 server(s), max=NO_LIMIT server(s), master is running


          Next we get loads these regions being assigend:
           

          2024-05-06 22:17:08,614 INFO [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=bd9b9493492af8b42a99ff7e4cee8eb8, regionState=OPEN, lastHost=l230-n2.<DOMAIN_G>,16020,1714828981071, regionLocation=l230-n2.<DOMAIN_G>,16020,1714828981071, openSeqNum=531452
          2024-05-06 22:17:08,615 INFO [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=de38ce1846144c1eb843ca9248efb791, regionState=OPEN, lastHost=l225-n2.<DOMAIN_G>,16020,1714828981130, regionLocation=l225-n2.<DOMAIN_G>,16020,1714828981130, openSeqNum=516441
          2024-05-06 22:17:08,615 INFO [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=6546461582f9e25f18f5c77f6cf8fe6d, regionState=OPEN, lastHost=l225-n2.<DOMAIN_G>,16020,1714828981130, regionLocation=l225-n2.<DOMAIN_G>,16020,1714828981130, openSeqNum=403570
          ...

           

          Translate
            MrNicen
            MrNicenAuthor
            Silver ⭐️⭐️⭐️
            May 7, 2024

            The following ones are odd:

            2024-05-06 22:17:08,626 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=audit:dzs, region=3da7e18863977749267f48437295889d

            ...

            2024-05-06 22:17:09,760 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_13>:<TABLE_46>,
            region=0486a586e762a8fc837ad4cf3f48c0a1

            ...

            2024-05-06 22:17:09,763 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l227-n1.<DOMAIN_G>,16020,1714828981420, table=<NAMESPACE_13>:<TABLE_63>, region=8b12508d2ef09a011875540b383135dc

            ...

            2024-05-06 22:17:09,931 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l228-n1.<DOMAIN_G>,16020,1714828981415, table=<NAMESPACE_17>:<TABLE_46>, region=fcf92fdae49288267f5b94f80bafd186

            ...

            2024-05-06 22:17:10,046 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l227-n1.<DOMAIN_G>,16020,1714828981420, table=<NAMESPACE_19>:<TABLE_12>, region=7766e1e93ed788e0233d1286968fc132

            ...

            2024-05-06 22:17:10,046 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_19>:<TABLE_12>, region=969d766f8ecaf8074572a97dd7a4ad89

            ...

            2024-05-06 22:17:10,047 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=<NAMESPACE_19>:<TABLE_12>, region=2d84f975f555ee262c64618055075ccf

            ...

            2024-05-06 22:17:10,047 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l229-n2.<DOMAIN_G>,16020,1714828981440, table=<NAMESPACE_19>:<TABLE_12>, region=2f1f50ee36b7d30b9f234380a02cd154

            2024-05-06 22:17:10,081 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=<NAMESPACE_19>:<TABLE_41>, region=dcf0f06facb516b3a05803f1510d35ea

            2024-05-06 22:17:10,082 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l228-n1.<DOMAIN_G>,16020,1714828981415, table=<NAMESPACE_19>:<TABLE_41>, region=ba1f621709cf61c2c7ac523765f11a2d

            2024-05-06 22:17:10,214 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_21>:<TABLE_41>, region=38e9a26a5df320c95349afc7ca374604

            These keep repeating

            2024-05-06 22:17:09,691 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740
            2024-05-06 22:17:09,691 WARN [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628


            After assignment.RegionStateStore finishes this first round of assignments in [Thread-16]:

            2024-05-06 22:17:11,444 INFO [Thread-16] assignment.AssignmentManager: Number of RegionServers=14
            2024-05-06 22:17:11,459 INFO [Thread-16] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1715026691459
            2024-05-06 22:17:11,460 INFO [Thread-16] assignment.AssignmentManager: Joined the cluster in 3.0330sec


            Next, we arrive at messages from master.SplitLogManager

            2024-05-06 22:17:11,656 INFO [PEWorker-1] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting is empty dir, no logs to split
            2024-05-06 22:17:11,661 INFO [PEWorker-1] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting] for [l229-n1.<DOMAIN_G>,16020,1714828981427]
            2024-05-06 22:17:11,663 INFO [PEWorker-4] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting is empty dir, no logs to split
            2024-05-06 22:17:11,663 INFO [PEWorker-4] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting] for [l227-n1.<DOMAIN_G>,16020,1714828981420]
            2024-05-06 22:17:11,664 INFO [PEWorker-1] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting] in 3ms
            2024-05-06 22:17:11,666 INFO [PEWorker-4] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting] in 3ms
            2024-05-06 22:17:11,667 INFO [PEWorker-16] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting is empty dir, no logs to split
            2024-05-06 22:17:11,667 INFO [PEWorker-16] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting] for [l224-n1.<DOMAIN_G>,16020,1714828981244]
            2024-05-06 22:17:11,669 INFO [PEWorker-16] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting] in 2ms
            2024-05-06 22:17:11,690 INFO [PEWorker-11] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting is empty dir, no logs to split
            2024-05-06 22:17:11,690 INFO [PEWorker-11] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting] for [l226-n1.<DOMAIN_G>,16020,1714829249771]
            2024-05-06 22:17:11,693 INFO [PEWorker-11] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting] in 3ms
            ...


            and after this follow the long lines from the procedure2.ProcedureExecutor

            2024-05-06 22:17:11,837 INFO [PEWorker-16] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79}, {pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9}, {pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03}, {pid=19, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:zbl_persistent, region=ba37bf35a4430884d1a6ebbca95a8387}, {pid=20, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:grh_persistent, region=68531e269838537d534e81683e2a7097}, {pid=21, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_28>, region=72f61b803e206cc95dc2a05b1112d39f}, {pid=22, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_5>, region=f696c18f229b7b6525405d55afdd57cb}, {pid=23, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:grh_client_timeline, region=285ddc10b5d812318af616f7986ac6d1}, {pid=24, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_51>, region=f56d6589a0660ea69432e3ff1bfa6941}, {pid=25, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=0a0f5b07562c6220f4b15c8128748925}, {pid=26, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_15>, region=d4568f96d953bd370840c8942c263d6d}, {pid=27, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_12>, region=7b8c2748761fe9da697b565bd51c29ff}, {pid=28, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_13>, region=911abd94189cd5cacad87054333c7756}, {pid=29, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=d028ce2dae91e85c853dfbf24167b89f}, {pid=30, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=907ba360d0a32358465ffa19b877d0ff}, {pid=31, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_6>, region=07c3225b0f58be3adfe3e10f4054e5e8}, {pid=32, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_31>, region=eba2fd53661e007937008e1b4b8723a2}, {pid=33, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=96d4c009fb28c87f11371273343c86c2}, {pid=34, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_53>, region=a1cc9bb5815382500ebc1f8ff2fe2889}, {pid=35, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_45>, region=72af38e5137228c4a96c5afd1483141b}, {pid=36, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_72>, region=2ae51a84d97bc465eca9b9d534585752}, {pid=37, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:tbb_persistent, region=60f28a92e81332855bc0bc8814b43c9c}, {pid=38, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=1041abc1f0324a01072e30ac4651068f}, {pid=39, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:grh, region=913fc5fbecc382c9eb130bbca997a52a}, {pid=40, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:idp, region=52b26bbcb3470e2fda96a27cfb2bc728}, {pid=41, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:nbr, region=9d53deeb6951b5625c714f52f40cc2e2}, {pid=42, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:ztw, region=3204e2b67fcf1ff91580c11f919c9261}, {pid=43, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_32>, region=f9459615516226d166054c69348c614c}, {pid=44, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_46>, region=cb9fe991b45b07aaca74b1b19cd1d8a9}, {pid=45, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_24>, region=3c0ac9e61eb863ac6834645a45b2ce92}, {pid=46, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>: Employee, region=3152b0b94f3dab7872292d320681192f}, {pid=47, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_29>, region=2c1fabd4bedd491f1d74f814ca788c24}, {pid=48, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=698bf8a9ad8970d1059273622b3beea9}, {pid=49, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=7a54ab62ca00389083cacfd18859b19b}, {pid=50, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:ijs_persistent, region=9ce8fccc5665f0256fb8ab8e25618bcc}, {pid=51, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:ijs_persistent, region=fa3effb1e5c14db984279936ab111895}, {pid=52, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=53d819a79ac851e839f5ff6a6b149b19}, {pid=53, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_15>, region=5864381b7aedceae4ffae2d1eb84620e}, {pid=54, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_51>, region=39dc5b21757fb3745c703656d2385132}, {pid=55, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_63>, region=484aacabb3202a69d985c8316db6ed2e}, {pid=56, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=5aa11c337c3f916a35a27dea488e22d6}, {pid=57, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=971001559dbb1da0c4518773a5bd598f}, {pid=58, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_72>, region=7d8e127283f4c0116ac780ad7274a419}, {pid=59, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_33>, region=145a675885029dd31097cbd697243703}, {pid=60, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_41>, region=4639b130e351fc822f26948d88ee50e1}, {pid=61, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=3e29eed2edc1b2abb1dbe7e416087835}, {pid=62, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_7>, region=815ce30ee31cc91f4a6ae4b1ff787bad}] 2024-05-06 22:17:11,885 INFO [PEWorker-11] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=63, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:wlt_by_query, region=85d79840d75bd9d605e7f3fa9a330655}, {pid=64, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:wlt_quantifier_index, region=9372d2b0f3629ed5c088dfc0aa541961}]*no* further _formatting_ is done here



            And first procedures reporting results?

            2024-05-06 22:17:11,887 INFO [PEWorker-8] procedure.MasterProcedureScheduler: Took xlock for pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9
            2024-05-06 22:17:11,889 INFO [PEWorker-7] procedure.MasterProcedureScheduler: Took xlock for pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79

            Translate
              MrNicen
              MrNicenAuthor
              Silver ⭐️⭐️⭐️
              May 7, 2024

              More long ones:

              2024-05-06 22:17:11,944 INFO [PEWorker-14] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=65, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_38>, region=d26d9da451ba6ee5d917d0e135f5a7bf}, {pid=66, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_2>, region=ac46967db9dfca5c91369164ac1c9469}, {pid=67, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=8ea10f9305a566d8efadbfcb2c6d87de}, {pid=68, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_37>, region=c038bc8eadb331b104131184d88316ce}, {pid=69, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=d2df8e2457a319244cfa85d898773f53}, {pid=70, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=c32fbee54febb51b1278c139729c9991}, {pid=71, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_61>, region=85a1c4f8c9303a693d2b0baa3f04e576}, {pid=72, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_43>, region=cf3a8c84ee842791b876765071465d98}, {pid=73, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_26>, region=0bb135123341acc459bc2b0e095c571d}, {pid=74, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_61>, region=60936d1899edbaae0788c2d928e34be0}, {pid=75, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=c3763274a6c84f17cafbb831c1a81132}, {pid=76, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=9f4bb127925ef66b2e2f181ee0b69d02}, {pid=77, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_14>, region=846e8ddaec0fffcb97d44480c7cc8798}, {pid=78, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_63>, region=cb50cf460205d12c5bee0e2fb1f4c21c}, {pid=79, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_73>, region=09b701e10fd060bf0901ff8ce55778e4}, {pid=80, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=a228e68c6d231c7d37a15c1c048bef58}, {pid=81, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_17>, region=7ba59a23baf79edc23ebbc5b152936a3}, {pid=82, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_6>, region=e94e891f60a0a0c194700f73504ab4a1}, {pid=83, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_54>, region=be8f1361b364a21f20300aee3c174464}, {pid=84, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_43>, region=1ee1d509aa4c312c6d48e2bcce1d14a2}, {pid=85, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=1621fa407abeeb4c112638af74553342}, {pid=86, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=530c9d659a0416d0f442a09e8070d106}, {pid=87, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=bd644295d91e525001cb1c08f82c06cb}, {pid=88, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_46>, region=69c55317c0eaec7c118fa09308c34997}, {pid=89, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_41>, region=a882652e5d221f3fc56efc5214970553}, {pid=90, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_25>, region=cf45f56e80d4b860220db6589e55fa4b}, {pid=91, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=f6f18f4f98aae325d3aab52560128a2c}, {pid=92, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=bc4a3997c5f8dd993dd3079044bb9443}, {pid=93, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_51>, region=ab6f3e8d44332b4e78a0aae110b26603}, {pid=94, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_29>, region=a9ed6a68f9edc2c93bca66f146d24e85}, {pid=95, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_14>, region=358157917db43e1aab7c64bde66e11b8}, {pid=96, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_26>, region=1491a0474a21f04733065b84a7ea0690}, {pid=97, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_42>, region=143bd5d9cd709b8b346bb166bbce71c2}, {pid=98, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_44>, region=e5eba35899c2e90162b869d8e75a0c76}]
              2024-05-06 22:17:11,950 INFO [PEWorker-6] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=99, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_27>, region=fd8364d6adfd11711610317f468a518a}, {pid=100, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_21>, region=81be87773b6c3ffacc0a07c9da8d7dd4}, {pid=101, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=ec8d8cf7f7aaa6e9740a1c43a2942ff7}, {pid=102, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_39>, region=31434aa06033a0edf3cb394a7d64a15d}, {pid=103, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_46>, region=b78b117f1426c5ce0e528eb51ad3ec07}, {pid=104, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_9>, region=fe81d1f92167a6099eb5516668314e3a}, {pid=105, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_62>, region=7b1b806d6b9fcfe5ac2f1d54a7a6229a}, {pid=106, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_1>, region=5b4d78b09c99ac59d10c83fff9303cd0}, {pid=107, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_6>, region=40c174d887cbfab7da2ba1977a60fe94}, {pid=108, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_26>, region=f1bf3decd899563212c071e24508b97d}, {pid=109, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_timespan_index, region=8f794f8b6ac4f70770d0f80a5e9b02ce}, {pid=110, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=3609823c9e0c2bc9c7d65d716e290bb7}, {pid=111, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_23>, region=d004e125f9dfe02fbfc8df621d4c9878}, {pid=112, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_36>, region=a724e0984e13a5dc9160a68ab72f5e29}, {pid=113, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=ae445ff6e5c92bed212429c28bf2795d}, {pid=114, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_31>, region=5c169953e27e09c477c78bfefe9ed88a}, {pid=115, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_39>, region=5c512440cca3566741e6d53e3bb29eb1}, {pid=116, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_27>, region=cab78895641204b76abb7229418ac4d2}, {pid=117, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_reference, region=dd6daf6ff096794f8ebddc0d28049b6e}, {pid=118, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=36add998cd3ce2524203ac9d69906530}, {pid=119, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=587fec8406b9994f1981094ef7e7ebda}, {pid=120, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_54>, region=b92a93a355d6a1ad7ff11bf3e34fb9ee}, {pid=121, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_71>, region=ccc28d5406e44135221ce70fa580cb6b}, {pid=122, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_18>, region=f315381dc89c081c0ff999106ebd0f9e}, {pid=123, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_7>, region=3f3e045bdedf9708c9167aef64477bca}, {pid=124, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=6898c984f4ef6345ff9656f783d0decf}, {pid=125, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=85a72fc500b1d803a1c20e7dd312994a}, {pid=126, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=85df5947445cfa615d210549d4113544}, {pid=127, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=08a4d55e7f60d570e90e08bca932e274}, {pid=128, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=5e6e5a2254aaaa9cd6bcc07787bd7b68}, {pid=129, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=1ff9f8b3fbdd559236bfbf3341a881db}, {pid=130, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=ad8cd76e0a891016efa454b052a4d709}, {pid=131, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=4666ecb9e11eeb2a1a92a5db20e9fc77}, {pid=132, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5a47edc1544c02826a522a8806af7097}, {pid=133, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:wlt, region=5eb06b8067fdbccef433aa61685e2bf1}, {pid=134, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=358a62174fa70ce3e594ac48b65265da}, {pid=135, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_52>, region=a7c63347b3eb5dde263cbfdd65b77a69}, {pid=136, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=f9f52135ba20e1fe76cf371acab6bd00}, {pid=137, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=29431818e46baa2963d67e2635ee83dc}, {pid=138, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_27>:message-bus, region=7da31638b73cf866e7995b1c77cfb2a4}, {pid=139, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_46>, region=0ff03da8b9c98538bbe04d1512144c76}, {pid=140, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=275807cbabad3122b0baddbc1d7352f7}, {pid=141, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_57>, region=887d0888424b0e60060a9cc8659bd35f}, {pid=142, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=987b0dc7408d25aa411924236432d746}, {pid=143, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=f14a6fb3f90f81ee77388964344c2eba}, {pid=144, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=6e505ffbb8a41fd3ec5ffa5678063228}, {pid=145, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_60>, region=14ec3ec9cfd01b8172aac7c2b6f99a6d}, {pid=146, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_59>, region=9b9897f6f40f973796471ab12a7ba9ec}]
              2024-05-06 22:17:11,954 INFO [PEWorker-10] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=147, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=00d20f4dd79bcebe1ecfc93b370d1941}, {pid=148, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=b3f91f5c8ef858e6d15c5bb8f7a7cd26}, {pid=149, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:cds, region=ac189d86bf78b4b9a24c476663874eb7}, {pid=150, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe, region=606e3055bcdd795113d6fa794e1d9843}, {pid=151, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_13>, region=2fe2e69dc40e61d76960e2fb9d87731f}, {pid=152, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_46>, region=c78535817a6d3956701a99e89baf6ca7}, {pid=153, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=0cb990ba90c53422b422e6a2b172d4ff}, {pid=154, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_19>, region=1c5466966c1aa44b2191db632806e4d5}, {pid=155, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_10>, region=71c761063de6a50e16750ab15921dabd}, {pid=156, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=5234265075f572118fd896d57b27018c}, {pid=157, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:hgg_client_timeline, region=753d0f68de8e47864ecdf59bc6c3f084}, {pid=158, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_persistent, region=ebc1328fab65efd0d9314130ceac2b94}, {pid=159, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_by_query, region=8f4c816f5ad863cd7176defab9d8b979}, {pid=160, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_by_query, region=272a0150d5fb7edb81e1230a5d38db18}, {pid=161, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_46>, region=4dd22258f7282411a06566357e44e335}, {pid=162, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_43>, region=5344ff13fb3db964fe346099ccbb5ad4}, {pid=163, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_40>, region=e762e13ed49b02ae79ced188bb68c6cb}, {pid=164, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=4f8445c40820a61633833e2e028d7441}, {pid=165, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_28>, region=c1ed331e961eeeb2bb6fa9d50f73a040}, {pid=166, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=0e3f779f3590717ee562fc40ac26faf2}, {pid=167, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0133fc91fc6d7bef529682456d19a774}, {pid=168, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=9755e4208ced1b938c53017ed30b6c12}, {pid=169, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=cfafb084581d983eef636cd1c09be90d}, {pid=170, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_46>, region=09af2afbe817abcf95dc0cb8825ed9fd}, {pid=171, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_by_query, region=6ff424adf916a4bdfbc4c58ad897d3fc}, {pid=172, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_by_query, region=9c4c92c7246ca209d7a02dca9033f544}, {pid=173, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_27>, region=432de3f682b58a0745464c9335be837e}, {pid=174, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=334b11e67ce7f61b8165e14ad470c9cf}, {pid=175, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_1>, region=9e7c8d1219f6b4f094f5d372aee740c3}, {pid=176, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=aec330cf8c9899290b1cca3d8dbaf7f5}, {pid=177, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=ccd9e06e5f8fcfe9dde1b65133fe8839}, {pid=178, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=9c1eec8741efe2d89c0567484cd638f0}, {pid=179, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=1bbc996cfa734e0fc6c70bacfa5aa4da}, {pid=180, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=7b9fa4da04f481d3258d7fd57b72a354}, {pid=181, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=8e894c6a576efb759fba7cc4da6be1df}, {pid=182, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5cd6925ee92449240ce14a37b275330f}, {pid=183, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_37>, region=48231aab340dff0874a068d75e53fa5e}, {pid=184, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_26>, region=3608dbe215817b2047c73be91604bfdf}, {pid=185, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_by_query, region=5c874fc02cd78b2a2da75bb156789c4e}, {pid=186, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_by_query, region=47211257f5f82a304ffaf05a8942858a}, {pid=187, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_48>, region=b0b022264b64be0b9fe4dc849e1cf1b5}, {pid=188, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_3>, region=01f9fd1bb65e07860fa74ed03266ac9c}, {pid=189, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_quantifier_index, region=2c017dc1457c9390388cd5644d3e3569}, {pid=190, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=bb88b7509375a9e1caa80b2c75d42a6b}, {pid=191, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=7a1d6e6b9390f9be391a2aa246282557}, {pid=192, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=35ae73ec75648c6943cffa7fbfcf42f4}, {pid=193, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_33>, region=adf0fc266fef938f1fb10efb21580df9}, {pid=194, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=f19bade589507b3931911a7a99c2f264}, {pid=195, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_13>, region=c53bd3f2a33d92f039ba4bb16efa518c}, {pid=196, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_58>, region=f3ceda41c170dc8ea5d6b2ef1a057b62}, {pid=197, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=769229ad8d8e8b8498974d2a643ea10b}, {pid=198, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=58ea387d639ee0f764ba90310584c9d4}, {pid=199, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_54>, region=23e746b75b557c89f5f3bb3f0d0de4b2}, {pid=200, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_11>, region=48e2a9ad25715bf409c0ab8390f92144}]
              2024-05-06 22:17:11,966 INFO [PEWorker-13] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=201, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_15>, region=b5c4d477bba1dfbdcc2165c523bd7395}, {pid=202, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat, region=bbbc7f79a39a1c8a417a0a7f9054d1a7}, {pid=203, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_60>, region=09cdd7faf2a89a737d813d648e1b028f}, {pid=204, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_62>, region=18a0f41c5988a80a493e05cae42ddf0c}, {pid=205, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_25>, region=bcbdc1f2e315acd61ec815d69df8881a}, {pid=206, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=7ccd5f985123343cd540a4cbdc9a704d}, {pid=207, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_7>, region=b6031e76568526723101704dd52bfc3e}, {pid=208, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_9>, region=c54357dec4f34c820a50ca53852508f5}, {pid=209, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=d53818a4613c3655c8809b488dd13fe5}, {pid=210, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=1c56709a7ef1202b2ca3745e178c1ab3}, {pid=211, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_71>, region=4cc6da9099a4fb0cb217a1dba51ef6bd}, {pid=212, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_37>, region=a7f97c7aabc91f42af675c73f03b3546}, {pid=213, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_35>, region=463c29108637a39f2429745d39ff73e7}, {pid=214, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=25ac0b75eae580989551c7d8b3773732}, {pid=215, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=3bc0c1a61c45f68dcbd0af9fb496e7b4}, {pid=216, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_18>, region=e1a8fb3db5e9daa5e2f32d10a29046ca}, {pid=217, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_46>, region=b912321cbb0dcea3280a6015465d5f78}, {pid=218, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_21>, region=87cf47f804e40b1a52a77f5f39b3f184}, {pid=219, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=e17d2f7c9838590707239cc949ec0eeb}, {pid=220, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=4699b0b7032fde3b1736d5d445091add}, {pid=221, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=9fe9190ef7db10a90c2dcf62187eb737}, {pid=222, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_55>, region=f17c894285999f568dc2d13307c30f32}, {pid=223, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_50>, region=59b8a67b8af82dfbb08afa40e6245ec8}, {pid=224, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=be9a5a341a519c132b4daddbd9cd5804}, {pid=225, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=88fa6bfdc0ef4c45b1fe5eb22de6e6aa}, {pid=226, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=d4b53b21efe1bf9a4632dced64ee0d1a}, {pid=227, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=9abbedce0ecb7e64aaab441dfb867922}, {pid=228, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=7137884e41ce4bd388567ccd7c91ad9e}, {pid=229, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=b03a1ac196a2758819ab14bcc2d182b6}, {pid=230, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=abc3172931d7f48c6613a6cf8268ed5b}, {pid=231, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=265d2fedcc86f4101420ff639be21383}, {pid=232, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_46>, region=15d526f60ea517e2129bc373e63ddf11}, {pid=233, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_72>, region=4a074ba4dca8703f0a7ddd87fcd50bfe}, {pid=234, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_47>, region=f494d0b158b7c3dd986314e242d93770}, {pid=235, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_32>, region=3301fea1f596540b4c605a8afb2a83a6}, {pid=236, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_40>, region=d3160c06c71d5ec702d391222f21540a}, {pid=237, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_73>, region=f5085d50d9508c1b11a3e0ed7e98c007}, {pid=238, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_50>, region=933660db79d8c4c42d9cc864493b7d5b}, {pid=239, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_72>, region=714388e2805cc386d7c48850b29e89a8}, {pid=240, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_16>, region=bef909fc3b5fe93a466007e73f8f79ed}, {pid=241, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_33>, region=f445d91aea86819a2fb4278d5448e178}, {pid=242, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_persistent, region=6d22694b0c619bbdd0b8ffe7cfa2ee73}, {pid=243, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_persistent, region=27729d98bbeaa6169d23642af41bb007}, {pid=244, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.app_flow, region=5e934d2af0695112564acfc62766a171}, {pid=245, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_22>, region=b2e955a6ea739dfe1ad4880b40e12d1d}, {pid=246, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=a9339217627c55dd9e2fcc57f9e70fbd}, {pid=247, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=512394fd78cc3fd00e72796f5e468f82}, {pid=248, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=0b90a930e8da2c15cf3765574a0ac85a}, {pid=249, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=a6fa48fac62e74f6ce14d81ac2a916d8}, {pid=250, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=6783fc26c108ac5d7e4c59c68d050ef7}, {pid=251, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_60>, region=df6752fb754f6a12769c9f8f6868f5fa}, {pid=252, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_42>, region=c7e68ad2b83cf444a3fe55c597abef1c}, {pid=253, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_9>, region=55ca2633ba2ddee3b6efa7e1b477c60d}, {pid=254, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys_locations, region=17ad83a7769cc356132351b13de1d212}, {pid=255, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_61>, region=04761c39e35764683291abfed1bb32d8}, {pid=256, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_22>, region=48750756b3a6e4414384853246bc646d}, {pid=257, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_66>, region=c8be351bcf2f0d0d119c560d1bc46f1c}, {pid=258, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_17>, region=b8f04ac037145bdf82440c6fde20d8da}, {pid=259, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_23>, region=3dbac8b33c33379e1812b12b6a148d32}, {pid=260, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_2>, region=9ca449153035cf901abd8402d29bf318}]
              2024-05-06 22:17:11,990 INFO [PEWorker-16] procedure.MasterProcedureScheduler: Took xlock for pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03
              2024-05-06 22:17:11,993 INFO [PEWorker-9] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=448, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl, region=52232b406176da8242980a3735e7b551}]
              2024-05-06 22:17:11,991 INFO [PEWorker-1] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=261, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_55>, region=9cabdbf8353308c257cf64514c0bfe69}, {pid=262, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_58>, region=fb87ff575b36c5997d8713378e47f89e}, {pid=263, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=d230357193249bbb002f0785f4c7271a}, {pid=264, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=00af8a539d6212c37c174562639d25b0}, {pid=265, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_63>, region=dba7407a60581148a46c4a135932f335}, {pid=266, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:cin, region=4ee39571768335fe74c62626ef424ec5}, {pid=267, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ztw, region=fa12c00dd61aab37437a09f92f9ef8f0}, {pid=268, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_28>, region=d8dabdd9f1b4080679de2e5cf24b53ee}, {pid=269, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=e7eb7c9c11ef47b3b3d77c0aebc6e978}, {pid=270, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=9672799e1d65bc522c12e4ae8443b412}, {pid=271, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=daaa022ec47e538c1bbdaedebbeefdbb}, {pid=272, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_1>, region=b03bd74027c4b06b0dc80451e5864952}, {pid=273, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=f567546ebf7fa1af2c3c1d913a6cec95}, {pid=274, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=ba777b929557fc60d1730984f6faaad4}, {pid=275, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=0692c4f2914cd80dfe661e3c6ba8ab27}, {pid=276, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_68>, region=2119a81b0ae71aec5dfe6a8635454016}, {pid=277, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_56>, region=f37c3c70820e5dd95166c6e0df6c266d}, {pid=278, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_41>, region=de842bc4866c150a1d0e02cf90743bdb}, {pid=279, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_41>, region=6a8808705fb01b06a9719d0263d4e119}, {pid=280, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_28>, region=4efcb1aa3e0f2cda75f0869656b69489}, {pid=281, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_34>, region=22be35f2c3c4d6024565c71803750012}, {pid=282, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=7dc8f2ceaef8ae01d52da98d3381be71}, {pid=283, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_41>, region=c190dcc9ab4138c6bdfcf4d2050954f4}, {pid=284, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_72>, region=4291fd7de0724af0dac6b5390f01b832}, {pid=285, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=eda5d4393ccb8a30964e057cf1585638}, {pid=286, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_28>, region=5a652c833612877e99cd17a18d16e25c}, {pid=287, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_71>, region=af5b7441e4dcc6be7a729a010f640b98}, {pid=288, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=878f434511529251fb80bd0fa615397a}, {pid=289, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=752a7e93a236de68fd172255e8dc59b8}, {pid=290, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=98aa87b3d0b14083769c178732e65521}, {pid=291, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_23>, region=fccde6c1bde39b9f50756232422bc353}, {pid=292, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_3>, region=10ecf35cb2bbe68c17f7bfc232e6ac9e}, {pid=293, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=81aa44df8d1be4ee7aa7b4b21506cb6d}, {pid=294, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=ebb3848ed489a7d12e74033fae54269d}, {pid=295, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_persistent, region=715fc0851a4a80f9570d15f9f42eeaed}, {pid=296, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_55>, region=c3466fc1a892f1cd76f93d31276b55b1}, {pid=297, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_19>, region=b8a6f89dd367215c3661ea2ad312eb2e}, {pid=298, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=25e33c575daa89c42bc891bc3133a438}, {pid=299, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_47>, region=da4740e4384c819e8b4475aafa668d23}, {pid=300, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_21>, region=5acae61fe9dc5ffb5d6d72e4fefe9062}, {pid=301, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_16>, region=16e4f4914e1b1df0ba72f5853fa525d9}, {pid=302, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=auth:account_activity, region=ed108e6facb210154c031fc595d2986f}, {pid=303, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=5fb1730f3bdbac7016cf22b70b5ebb27}, {pid=304, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=f8fca7a15afbc527020fad926c0daead}, {pid=305, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_37>, region=cc19878cdcfadaab0aa41bccffd1bd65}, {pid=306, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_26>, region=0c0437c6e51070e2a1eb57386c3f2bb0}, {pid=307, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_1>, region=7123221617b10eb342325353513d7455}, {pid=308, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_12>, region=86ccef773c3fd228453a37c170aece84}, {pid=309, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt_reference, region=d3534603c564eda83632939a3fa1224c}, {pid=310, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_27>, region=f151a2ac50ca31cfcc64b46f6859fcd5}, {pid=311, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_27>, region=5cfb990ab8940b24cf7f007e7ef0bf7b}, {pid=312, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=5306405ccd189c1ef2d5962da914349d}, {pid=313, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=90ce8e8e81f751303cae2a5772c21795}, {pid=314, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=89a247d379f35e764b0b22504b78f325}, {pid=315, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_48>, region=bdeca0a3f7f9e5b7c9937e6e8a1d1be4}, {pid=316, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=7e3532e48f693d8d289d813148d6a4f5}, {pid=317, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_56>, region=4ab6f8e662514eba09398bcc767febaa}, {pid=318, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_28>, region=a29c8fc117e40eac93e0f9e119553614}, {pid=319, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=3da669b0bdc76c87363281b8e99dfa8d}, {pid=320, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=e57f5bde53018a69c7c0022493fcd354}, {pid=321, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=1192336a35ab7cb2fd8f575cec43fff0}, {pid=322, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_10>, region=278fbd8a05b5d09cb4cb4b409c864584}, {pid=323, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_60>, region=be429de23e05dd59f824f0e12ed4c4d0}, {pid=324, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_25>, region=b7cdce4d0b956eaa69a766672b3090a9}, {pid=325, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=bc8adad0a53341adf39cce1f5bcf3e9d}, {pid=326, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_25>, region=89b61005b4de954cf1bd8a1ed046c294}, {pid=327, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_3>, region=c2890cd6fe28924884b902c60f6ff434}, {pid=328, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_3>, region=d43cb6cb67cb7ae0ddb3a577bfedb707}, {pid=329, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_54>, region=348826fe55734025419e00d13f178218}, {pid=330, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_60>, region=76a52c147a6b738fee00abf0a8d38cfe}, {pid=331, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0de76b556fbaa3bd1a3aa02dcb9c3a3f}, {pid=332, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0238767c59208b1368ec21218f7361ac}, {pid=333, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_49>, region=707c27be94d45223eaeb876b0929793f}, {pid=334, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=bea7ac8a617a7380cd5c2545e420d3ba}, {pid=335, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=c51df52ff432403504fcb9c5b650290c}, {pid=336, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_40>, region=d94808418d2327e2e61a3d9cd141cc5a}, {pid=337, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_11>, region=fe4ce65f39cffa3fa363e07ad51a94b2}, {pid=338, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_5>, region=1a705b0e8eeb0d1f9561ea34a7d2f7f8}, {pid=339, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=4aa2b82b3718192d0c0126855ff9cb05}, {pid=340, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_46>, region=c3e304de319ea07e830463cb8212bcca}, {pid=341, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_57>, region=0a179c22a5ce6f8c93fc3cefe4a1f864}, {pid=342, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_60>, region=0b3ef2bc9d82cc296fe41b46de26ed78}, {pid=343, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_7>, region=0deca40e0bf3442b4c73726b16e19743}, {pid=344, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_12>, region=b541f926df73acfada2e2a35642d7c07}, {pid=345, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_73>, region=aff63924d0be9e25291bdba5d2bcaab1}, {pid=346, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_15>, region=691aab81768ff6a808b605f7532cbd1c}, {pid=347, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_35>, region=8cf9e7c507961c6b6cfd5e09397e274f}, {pid=348, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_34>, region=1e1ab991cd7fc4fcea21491fc4271e50}, {pid=349, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_43>, region=3deadc750ca47a175e7bef12b9396233}, {pid=350, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_55>, region=8bc319cfb1232f5e9d8cdab58fdb73f5}, {pid=351, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_1>, region=ff435f619d4ebca70dccea01c1544419}, {pid=352, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_32>, region=5ee934b10ba0d0a3ad5187e8fc4ebc6a}, {pid=353, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=28e539bfa8ad97ccdd163eccbf272a29}, {pid=354, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_22>, region=44bb488a9e65a184488146ffff903e65}, {pid=355, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=dc9d9c2f14473333d6d05754356d9ac6}, {pid=356, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=1f3a3259f905d22af35549c2db081544}, {pid=357, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_11>, region=c8dc2b7ba47fdd1b075dca81cebdb645}, {pid=358, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_29>, region=510e50899e5fb5e8ccfb71ff63c5638f}, {pid=359, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=989ca9c8a3f5a1756ccdc1089fcc4cc1}, {pid=360, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_6>, region=e8937927d7af6ee8273fdae233beedaf}, {pid=361, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_13>, region=93c56196d1938a5f4136567602317244}, {pid=362, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=8986a928fe5c5e4325198664ce5d17fa}, {pid=363, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:cla, region=8402fe5144954dc56d45adadf67b9cbf}, {pid=364, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=d70ca2c40b9ffc76d58e7a530789a8c0}, {pid=365, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=138cbafeceebde3459edf79fda73d450}, {pid=366, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=cc58fc9fedae3fc75c8655986902ba6c}, {pid=367, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=479d22ee3490e4191ef1d43c17fda93b}, {pid=368, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=5febafd6d035f8e20dcde9a4f74e61ce}, {pid=369, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=b162fa7207b2a21cd616522946bfab3e}, {pid=370, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=4f968e6ede745833d6c639d0a9ff2807}, {pid=371, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=9e4c23ea1ce446781c57a72d59ff75a1}, {pid=372, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=f4e41961f3920e038f0a5aec72a7e667}, {pid=373, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=d382dff4e9eab83e22dfd63943168677}, {pid=374, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:wlt, region=27021a1dd9b3a3cf0ac8d9022746150e}, {pid=375, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zao, region=4f3f5a5bf17be1aa79a03e94c1285576}, {pid=376, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zbl, region=9a948d0c445d3d810f402b9cf77f8a85}, {pid=377, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=bea13c1ef47833992db43210e38ba678}, {pid=378, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_61>, region=816769a8ff9c8d34a484441ef39c2b1a}, {pid=379, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_15>, region=f2d1d8517f4541a4916db74e63e9df0f}, {pid=380, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_70>, region=c6efd6b6836e968552d8f531ff1358c9}, {pid=381, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_25>, region=2e74a82a4b3f6297076ef6e25261b460}, {pid=382, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_7>, region=56666ded0297dd0136797f4844b33a0d}, {pid=383, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=eed5ba2cf6b68aa4ff6149759223c266}, {pid=384, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=e04ad9181d3577dac7c126c66c721650}, {pid=385, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_15>, region=8bc9b9261ecb41b744ae8fcabae8bdfc}, {pid=386, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_65>, region=ee223a2d3c84bd2a35227124ccd3efb8}, {pid=387, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_8>, region=8f388296a24cf746ac8762a16698ca22}, {pid=388, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_58>, region=f86e25e7ca220db5c908669b3de1b1f3}, {pid=389, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_42>, region=05219a664defb595ca7658a1bd7d511a}, {pid=390, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_34>, region=ee0bc298e2fc3b8dd0a15fec2ae894d6}, {pid=391, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_23>, region=96eff57175c090f342530b2f41b2de99}, {pid=392, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_23>, region=c8020dff04475030b1ca6d9fb66475e0}, {pid=393, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_62>, region=9fcc9a48a801657d7abea81a6f41b3b4}, {pid=394, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_9>, region=5e39e64343d157bca285adf8cd1b4753}, {pid=395, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_12>, region=7695f48743f8d50e36b783ba93ce5bd7}, {pid=396, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_53>, region=4265434f477b4eadec5918f00851ea86}, {pid=397, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_72>, region=91c5fa07f5bd2297ea2b1c3837b69407}, {pid=398, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=3f78b16536cf86845f913b6d333d799b}, {pid=399, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.app_flow, region=662005b2e8162740152cdca4d80ab677}, {pid=400, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_14>, region=3b9fe4b67b5a48ca8e5c6ac5fb7006bf}, {pid=401, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_64>, region=ee909bbd56543c5191161088fd5a806c}, {pid=402, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_63>, region=d6d13581484e69ca42f4702a1a6af9f0}, {pid=403, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_63>, region=5d854f1eb7bcda240d0f6d6bb8545815}, {pid=404, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=9d9ccd2bc40a4c358e41a86ee9957c15}, {pid=405, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=00c8319a210d90772fd031a45c88ec36}, {pid=406, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=f3a773f2add03d4e976fa8a5ab200c0c}, {pid=407, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_51>, region=3bf60504f9714870997e24ed142d3faa}, {pid=408, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_72>, region=4b2cca0a0cf2723a8488e88bafacf95e}, {pid=409, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:services, region=07466e00c206b9b934417f0040059a67}, {pid=410, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_22>, region=c3d9ee32a04d228846ea0bee462f4e55}, {pid=411, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_26>, region=abc6e8e483eb33ef88e9026e6ba8dfed}, {pid=412, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=73ef17461282d50cff567f089e432bac}, {pid=413, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=8e92a16af113fd860455d91f37aae3b7}, {pid=414, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_11>, region=16fbec1044017e15b5b3b514b17eb59a}, {pid=415, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_6>, region=6ad83e8d695932a4075f6213fc924e07}, {pid=416, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_24>, region=7ba8496b2e2774e476521870581eb137}, {pid=417, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_72>, region=77afca33126ac4a4d3650471e4f51852}, {pid=418, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_4>, region=4b3368505a01a7db9df97335fa2c24b8}, {pid=419, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_31>, region=c942b0795588e8965cfadeb8b409f510}, {pid=420, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_40>, region=15e502c41f2b932092c32a8613f4948e}, {pid=421, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_23>, region=0d51be0e6f91c2df9e50fb6bd888ae5b}, {pid=422, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_43>, region=e6eb36d4c9a022460a70b71cbb85038f}, {pid=423, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_70>, region=fb283affe3bde8f245524fc2a4465e66}, {pid=447, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_13>, region=3129677490b14ed8ce90f0c386f5af60}]
              2024-05-06 22:17:11,991 INFO [PEWorker-7] assignment.AssignProcedure: Starting pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null
              2024-05-06 22:17:11,990 INFO [PEWorker-8] assignment.AssignProcedure: Starting pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null
              2024-05-06 22:17:12,001 INFO [PEWorker-15] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=449, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr, region=2be9537800624651fdc23200b9d052e6}, {pid=450, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt, region=6d268b7ba4e5a74ddb13a99f210b7aa9}]
              2024-05-06 22:17:12,038 INFO [PEWorker-4] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=451, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=c61efe7f834bd5e8fb1ef573c6476027}, {pid=452, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=466adde637ee5d7641d5c2105074b9f4}, {pid=453, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_32>, region=3917850b8d9ad87fe38019f012fe2add}, {pid=454, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_41>, region=eae44441062d5dc8c0688b8f055e60a0}, {pid=455, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_41>, region=46707cfcb32eb52dbe9e8c8203afa443}, {pid=456, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_46>, region=92d3626516835796829a9a45c5956c4b}, {pid=457, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_8>, region=b2bb4b7465cdcfebe64237094ce948d2}, {pid=458, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ztw_locations, region=6adeced16672c2f98bc3c1aca6e55db0}, {pid=459, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:hgg, region=9330c212add67cbecd0e4bbb65f72178}, {pid=460, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys, region=7afb999201aeeea87e994597952c197a}, {pid=461, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_11>, region=1f897c1321e379b023db8dcb955005e0}, {pid=462, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_66>, region=e694b978e0c122633fd837284053f61b}, {pid=463, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_63>, region=30a70ae706e91a222274166de838111a}, {pid=464, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_63>, region=83a74451025580bb7f9d0cd11d10a139}, {pid=465, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=6d3586ba82d879a8041a830e463cee79}, {pid=466, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=1c46655faa39c4ab57339fae984cfbc3}, {pid=467, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_61>, region=53e0297c25320786bb23231acb2b6b04}, {pid=468, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=cacc72e20e62f4067f47519ad1d267ba}, {pid=469, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=a7c7ae317038d7a576618d3fcd27812f}, {pid=470, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_63>, region=b8100430c8dce21a0d29651e1cdfa327}, {pid=471, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_66>, region=a94e776a27155a45f4ae995a595598df}, {pid=472, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_19>, region=f708a1b93798a3a32f57ac88298f44f3}, {pid=473, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_7>, region=8680fb202f5002291d01e33f6ae0c677}, {pid=474, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_64>, region=cb2f8e4706d0671b296cbfa6dd64d80f}, {pid=475, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_39>, region=827abc98319f1cb36091de23bb8d0c4a}, {pid=476, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=e7027e77bc5a3502b93817fe70f5b1a0}, {pid=477, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=f84659ec3a9df6d12f266e23d77dfdbb}, {pid=478, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys_quantifier_index, region=9585cb481dab76c8289ae37a2070e733}, {pid=479, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_2>, region=73ad4e436fe8a4c454ad8cebbb5229dc}, {pid=480, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_3>:<TABLE_5>, region=a5464cc09a771f2bf2eac2d724cc2fea}, {pid=481, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=c1e760cf795e36d5c930114b7a3b6c69}, {pid=482, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=83f8632fc23f1acbf2c1e9766a75b782}, {pid=483, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_66>, region=cc3f02439ba1b94dad139118a39e7fc4}, {pid=484, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=559e67f55cbb06b0130690c943b809ca}, {pid=485, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_30>, region=913134de6319bd6745c9b89c4fc187dd}, {pid=486, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_63>, region=fca99618699b16e3cbecc721c923f547}, {pid=487, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_66>, region=97ec5bc08b8ee3e9b0465f34aaa1b686}, {pid=488, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=e16719230d3f0e3f93ea2812c58b5be1}, {pid=489, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=ce4aba20a0fc64b3a5cb7875bcca9bb0}, {pid=490, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=8b87f38f2951c4ea3c61d07277b7b896}, {pid=491, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_29>, region=174010da7a14cb23ed2d4cda5066f614}, {pid=492, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_1>, region=abbe0ec0457aab7b4153e6f2fd06fa85}, {pid=493, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_39>, region=1116fa6dd513258f3d5affa052962bfa}, {pid=494, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_38>, region=bb186abfb55ebdcc607b27d71668e200}, {pid=495, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_66>, region=e837a6cdeb7ef4c0145ed521200a52e2}, {pid=496, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_67>, region=7c97ccf6990a6eaea378421b8c577e3d}, {pid=497, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_18>, region=4f857594b08360c5522bc4a3d4097997}, {pid=498, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_12>, region=c6bdd90b36bd98a20781219b6af2d303}, {pid=499, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_65>, region=ef6516fb5fe98a69df64db8724b3517b}, {pid=500, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_4>, region=9a8fb840b2caadcb1c179474eaa635b1}, {pid=501, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_42>, region=0a1b3f8f3d99e378328245963da407cc}, {pid=502, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_timespan_index, region=4bbf2884d5af0d06f627ad9d193837d0}, {pid=503, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_26>, region=35c34d62af1e731f278c45961a307836}, {pid=504, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_23>, region=abf3962c01cd35820afc1399f4d6a405}, {pid=505, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_18>, region=ff1798effe6fc9d1f37cedf02e58e798}, {pid=506, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_61>, region=bf08e209b6e07121851d274a628fb856}, {pid=507, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_38>, region=f812525ecdd82c440b8dd8d5e958e2ec}, {pid=508, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_50>, region=e8e3c6220bef865656c4a6d568ca8008}, {pid=509, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=fb8af02c18de3e29eb27f5f1721e4f7a}, {pid=510, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_72>, region=9ee442a3b9ae69e207bcc6411cb72c58}, {pid=511, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_42>, region=480f99176d21b082375a33fe3bf12d89}, {pid=512, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=702cd74f208b5cffd27d45b0177634ba}, {pid=513, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_11>, region=8335322953acc00d697149fa02bc039f}, {pid=514, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_27>:<TABLE_69>, region=66136ae9052fa89bef76c42bef64c009}, {pid=515, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_24>, region=399e943e9954806aaef8d6a2bc70e1fe}, {pid=516, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=59e65b38b13dba3ea251a733ff328e59}, {pid=517, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_59>, region=ca8753bebe73cc6f6d5c664d619c2ae3}, {pid=518, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_27>, region=767032840c65e8f9cd5089a22ee1643b}, {pid=519, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=8d5801804805600bea72ce5cd9446c48}, {pid=520, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_55>, region=bb8faf48d39cb9c5667c4f4b81e37388}, {pid=521, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_45>, region=758702557664af51f95e7892184a84c2}, {pid=522, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=b31f6d9cd79b77f7b387a48f1578f283}, {pid=523, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=85f527a85b1459ba4d0515cba9f33d4f}, {pid=524, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_49>, region=a501efad6e61a886accd69096a51de0f}, {pid=525, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_55>, region=d685720514e4b88309023b6a502966b0}, {pid=526, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_43>, region=49d9e7be19851aa294cc7cc4b55d9e40}, {pid=527, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zgb_persistent, region=8f72a863e1c3d5720cd3bbe6c7971976}, {pid=528, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_25>, region=25b787ba9ac7be4bfaf16d4d5a48b1c7}, {pid=529, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_72>, region=12cee6e8df27cb239ff357e0cce911f1}, {pid=530, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_26>, region=3667a7af201512c963c36928d0beec2e}, {pid=531, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_9>, region=a4ef2c4778117fd18df53fbe2db96e16}, {pid=532, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_27>, region=3e9a743da8b6729ebcf46c11dd9592c8}, {pid=533, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_42>, region=c0dda7faac9764443d415ace31bd457f}, {pid=534, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_6>, region=658d35c14ffcf80187acf4a7e588d8b5}, {pid=535, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_20>, region=3f8100c740920f820672a13942110ced}, {pid=536, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_53>, region=a8886ffa4fe8011cf3644aaa8686f5cf}, {pid=537, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_60>, region=79c7205c1a713ccb93057d36df90fa4a}, {pid=538, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=588b3fe66bd63bc7b9aec55554e6baf8}, {pid=539, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=5dd7d57f8858f77f59493d767577fbe6}, {pid=540, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=85e30b21a4d068e598b850cf405b5a09}, {pid=541, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=8d02e4646ea1e3b269b0df8667c49b5b}, {pid=542, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=24636244d51e67e3a35843d2b44b0e06}, {pid=543, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=2438f52750eed160df63520c76e04bc7}, {pid=544, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_19>, region=09a5df83079bf8b9e2f0ce3e05d35e9e}, {pid=545, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_27>, region=d3d4f6e36922bf18cf888da9e48daa8a}, {pid=546, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_6>, region=4e32e01434dfe469101bb291c037b861}, {pid=547, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_53>, region=9d63646dcf52a57e26314aafda41ac3b}, {pid=548, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_37>, region=ab3395aef65ed976125fe46e5a837d36}, {pid=549, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_71>, region=c32c2e3905359e69ca142395a088cf97}, {pid=550, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_11>, region=3bfbdc17c6f863c691e9795dacb68c4e}, {pid=551, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_34>, region=c1860525b46b5bd2f9586d2c87504820}, {pid=552, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=4358f469f4d74d50d66331ee74277e02}, {pid=553, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_43>, region=9ae4e344d28cf89d9781d9b3cbb2e73b}, {pid=554, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=0e0051bd2e1b2cfa869a623cdc909763}, {pid=555, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=c6b220e50e28a3b3bf4a5f29fab3f2cd}, {pid=556, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=92dfdd8a2532cb399e6e67eba5664a79}, {pid=557, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=0038e7ee32d606dc00bace075e151fc5}, {pid=558, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_24>, region=02bb3db34a8e992383401b41879a8ad6}, {pid=559, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=3e19e390336d24abb392ce5d53081cd1}, {pid=560, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=752425315bec32b835fd86bcc9e1d955}, {pid=561, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:cla, region=0e8e45dc138f0aafc891a7611c2b0cb7}, {pid=562, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=376d53328d29de238443148140de46a8}, {pid=563, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=b4c74b65072baf60835a410145b819c1}, {pid=564, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=d1194857a1fa4d9c9c2d57a2e190d5ff}, {pid=565, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=47b0efbd946486e7bc824ec5a9150aac}, {pid=566, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=c85e4a00d06036294f35f263269bda4f}, {pid=567, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=e6c0e6d7bbe6ad01be65e9fe57aba426}, {pid=568, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=cd8e266bde4dc549910a88146e070553}, {pid=569, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:hgg, region=89b70ce278744658a639eddf6d90f44d}, {pid=570, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=89ed47dfd54aaacce7298ed7ffa135dd}, {pid=571, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=510c369b6cf0c330a9415854efd36873}, {pid=572, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=4a49c9bac8263915c77b8b116f85e5f2}, {pid=573, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=f315c4987d7f8e96d35b5a5b836c18a0}, {pid=574, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=36546490b285ad7f21af0b19007480f7}, {pid=575, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=00cfea50f773a27134e67aac6dce8fa1}, {pid=576, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=0c4ffaf5aeca347a19cf9eed3ddc5358}, {pid=577, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=509d458e718f97e075e26d10be568e3b}, {pid=578, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=09db0b1c02f96a50d1fff02f1c8af7db}, {pid=579, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5e800770355a88f427488b629abb57cd}, {pid=580, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=83f7dc30a5f06d5adb13524588855818}, {pid=581, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=291b502a62b078b5586d649885f5d475}, {pid=582, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=db1e76f15e3d73523ebdaf1c1c06abb2}, {pid=583, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zao, region=38e95d30bba9428234fb15221dc6ce5f}, {pid=584, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_12>, region=9b1f61309b94ada8009a178ddb4e4636}, {pid=585, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_59>, region=8641d6490c9b56c408d94ac096740e3d}, {pid=586, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_46>, region=354f0c8a1370815286b51a4b1582d3f8}, {pid=587, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_14>, region=5b2dcd3b95616450baf72ee664188293}, {pid=588, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_29>, region=6d9ca2a6acfe4ac471cbccdc120c3b04}, {pid=589, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_24>, region=2e2b88b0bfa20453c49028eced20c2ee}, {pid=590, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_7>, region=6a8efda89913a2b556771c59c3047345}, {pid=591, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_27>, region=24c5ee2022b934078c0f78d21deff906}, {pid=592, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_reference, region=dc0291681b786728f6cd21ee6a23bb6f}, {pid=593, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_42>, region=342672c1865f212a3f0fe41dfe989b86}, {pid=594, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_61>, region=a56d1d3443e867c6f2e4d4204556589c}, {pid=595, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_54>, region=dbac96812aba03d24e549e0b89c8650d}, {pid=596, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_35>, region=3d41647bd0a155cfed283088a1179728}, {pid=597, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_34>, region=3415b85569434236da100aaae8fd01ca}, {pid=598, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_30>, region=d8582c530ca270bea5c3fb3bef040de3}, {pid=599, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_25>, region=2490192492c7652d71ddd69df4f8b8c1}, {pid=600, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=5d4fc7dc0f39106a648bcf47f1552680}, {pid=601, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=5d76b5dcd749c89e2ea2977711a06c7d}, {pid=602, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=6595469e6bb9dfe0e58100daad516e03}, {pid=603, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_31>, region=9ba2f6b08e1da4bd2d47a4df7a7032aa}, {pid=604, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_57>, region=8fcc91a518f6f34a036b6c5e6ee7f4c9}, {pid=605, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_34>, region=bf077734bf83bf98d2d1f078fa2dfcf5}, {pid=606, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_26>, region=d51433bc454a247671a6fc8d8cc34507}, {pid=607, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_61>, region=b8b53b12a6855eaaf23f2fcd86d80935}, {pid=608, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_65>, region=e6955787e6997e9f09d6e685031955ef}, {pid=609, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_71>, region=5560528e9a2a93d789b7be1fc22a2c16}, {pid=610, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_72>, region=fc009e8566c9b8db9fef4977d10f6cb5}, {pid=611, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_29>, region=788e6997db951c8ea7fb9d637f8487d8}, {pid=612, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_25>, region=ee67ec547647c28fdeed0101b8296681}, {pid=613, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=3e0baf03cc1879ce98f594480ffe4632}, {pid=614, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_59>, region=0f168557d185be24d8236b1f577e7c7b}]
              2024-05-06 22:17:12,053 INFO [PEWorker-5] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=669, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=75a9ecc30a2fd3276de9ac01c4a3b6af}, {pid=670, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=7e82b7202431c0772d57b96c859c815d}, {pid=671, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao, region=585a71127b4393c583f2b1a39536009e}, {pid=672, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=b077d9c6af116f153cfbbf156dde77f6}, {pid=673, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_57>, region=ed80d78492d5b36bd676daf7dff5aa26}, {pid=674, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=fc8f9b834ef15ada068f9054f0a3246e}, {pid=675, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=e14adc8133fa8e50791a7ea74c88b1ee}, {pid=676, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_60>, region=55c76fd6461d42a68ec583d1be33b43a}, {pid=677, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_53>, region=07baa87f9c9c904c72163ec6e2d9adb3}, {pid=678, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_43>, region=15de5b3bdebed284129ddaf35009b932}, {pid=679, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=ffe92559774b50614e7551275f6594dd}, {pid=680, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_67>, region=1b814c1cbf8c1e9de7565efd3b25f990}, {pid=681, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt_timespan_index, region=0adb337e4d98c90d5301d9964f4282ea}, {pid=682, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_5>, region=c9826b585da29b6840208c1662c03086}, {pid=683, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=d8a1bea07427eb558c39190f8f946746}, {pid=684, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_46>, region=7def6eddc52abcd19bab85839914a1a1}, {pid=685, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_71>, region=dc099e9ba7d0d3f5f9aa6be10b80028e}, {pid=686, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=8db2e5cada5820754b004acbc880ce17}, {pid=687, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=c799db86045094fd0320e5fb37323173}, {pid=688, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_14>, region=0bca31436f8ae42d4d06ef1dffb6b7fe}, {pid=689, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_by_query, region=9c7f4d8ef637f8e2fb0ead64bdcc5c36}, {pid=690, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_by_query, region=fd6b15de65bba3d43436505017e5b2ea}, {pid=691, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=713e83fb0544e3f507c4cfe1b9f510bf}, {pid=692, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0e79e0171eb7ed47389b4eb984fca553}, {pid=693, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=09ed5c26faffb87ca8c6843a0a7323ed}, {pid=694, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_8>, region=8e267c07f543e25bc2a320707a204d21}, {pid=695, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_10>, region=a362ff12ae734f09c9dad5e2cff7dddb}, {pid=696, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=77127d1dca307dc74ccae9e12a23078e}, {pid=697, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=8b4402597026de84717ff39705f490b5}, {pid=698, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=43c366fdcd5094a13209c7bae30b7a9c}, {pid=699, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=9469573427d2c76940adac29705267ae}, {pid=700, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=55617f37c3820f05aad9aef85e4d9346}, {pid=701, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=16e0ce258462e89603eb7697228ab6f7}, {pid=702, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=ecea80ddf358a89c803ee0f1a8cc8b5f}, {pid=703, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=1c4091e7e7cedafb58beea23cd2959c2}, {pid=704, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=50f17b259f03ce2f7bf7ebe0b41c5e45}, {pid=705, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=1d3569bdb8ae48721d91d0783e77cd92}, {pid=706, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_48>, region=b2a4ab25b11f009e6607d70bc9ffcf3e}, {pid=707, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_46>, region=dbdbc86496bb9c98e638323bbefeee5c}, {pid=708, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_56>, region=d41f6b1862d2ae11100d85565f8e5fec}, {pid=709, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_37>, region=7436bb293ecaffb269b22bf9af7c8576}, {pid=710, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_client_timeline, region=d8eaadd1083d3a03848396f2c8d8c8ef}, {pid=711, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=2fbd1ff98a1b2442e718c43ac48f4734}, {pid=712, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_66>, region=e06908bb602a88955af02e853dbe37df}, {pid=713, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_3>:<TABLE_31>, region=177abc45acf52c795b6dd6f67145c5d6}, {pid=714, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=9fa16bd1cdcd629bccd2f6ffd381a7e5}, {pid=715, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_28>, region=e94e6814d161dffd12834ae227ffef45}, {pid=716, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_63>, region=ff9ceda3a8b8bd17fc4d627476ba2255}, {pid=717, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_1>, region=301e73eb5950ae0b8b5ba2b97a30ca13}, {pid=718, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_53>, region=099257cfe303f0c20275b4a8e8c3e276}, {pid=719, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_62>, region=8172981d31c57437920ab13f6bea9a4c}, {pid=720, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_57>, region=59425dfd7861755acfcaf4a741a9e402}, {pid=721, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_26>, region=8d9d3eccb0b62fb9746166f35139169d}, {pid=722, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:demo, region=370539314e0bf6e2133520805e16e594}, {pid=723, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_41>, region=fca59829f9914d8bc0ef56065f18386e}, {pid=724, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_51>, region=00e62790dc24813dce28875ce8cd5b3f}, {pid=725, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_40>, region=0d9ed0f5f7fc63838db85dc1c406c54e}]
              2024-05-06 22:17:12,097 INFO [PEWorker-11] procedure.MasterProcedureScheduler: Took xlock for pid=19, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=ba37bf35a4430884d1a6ebbca95a8387
              2024-05-06 22:17:12,097 INFO [PEWorker-16] assignment.AssignProcedure: Starting pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null
              2024-05-06 22:17:12,099 INFO [PEWorker-14] procedure.MasterProcedureScheduler: Took xlock for pid=20, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=68531e269838537d534e81683e2a7097
              2024-05-06 22:17:12,100 INFO [PEWorker-6] procedure.MasterProcedureScheduler: Took xlock for pid=21, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_28>, region=72f61b803e206cc95dc2a05b1112d39f
              2024-05-06 22:17:12,100 INFO [PEWorker-10] procedure.MasterProcedureScheduler: Took xlock for pid=22, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_5>, region=f696c18f229b7b6525405d55afdd57cb


              Skipping INFO but keeping WARNs here. We have seen these before

              2024-05-06 22:17:20,363 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_1>:<TABLE_51>, region=ec4e4dfe2c2bcf372ab8185150d41e1c reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
              2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_4>:<TABLE_48>, region=7e3c25195bd235046fa37428a5a58eb7 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
              2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_5>:<TABLE_3>, region=ed1abdd5978b355bc547f3322bbc79bd reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
              2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_7>:<TABLE_28>, region=319217e06a1140bbf8d3e569e1931db3 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
              ...

               

              And more of them:

              2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_4>:<TABLE_33>, region=d41e7923f187ee13d081591227c7805c reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running
              2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_5>:<TABLE_40>, region=7cc3121f8462d670c938bd3844861436 reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running
              2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_8>:<TABLE_12>, region=2788b213d23851dc98611dfc43935e9e reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running
              2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_9>:<TABLE_68>, region=460607d7a84d9b03c58f853ecb5a2b73 reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running
              ...

               

              And much more of these:

              2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_4>:<TABLE_48>, region=7e3c25195bd235046fa37428a5a58eb7 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running
              2024-05-06 22:17:23,457 WARN [R...
              Translate
                MrNicen
                MrNicenAuthor
                Silver ⭐️⭐️⭐️
                May 10, 2024

                We tried decommissioning both busiest region servers, which had combined 80% of all regions. Decommsisioning fails because of the same Stuck In Transition. Luckily the side effect of decommissioning is all other regions did move to the remaining 12 servers, which makes that these two servers now have combined 3% of all regions. To our customers we have a much more balanced and therefor snappy cluster now. But as long as we have these Stuck In Transition, autobalancing will not happen, and at some point will have a imbalance again. 

                It seems that the cause is that these 134 regions remain in OPENING, and should become OPEN, which is not going to happen. What is the best approach to mitigate this? Should we use hbck2 to set the region state manually to OPEN?

                Translate
                  rki_
                  Gold ⭐️
                  rki_
                  Gold ⭐️
                  May 10, 2024

                  If the Region server reports that the region is already OPEN, try to scan the hbase:meta table from the hbase shell and check what state that region is in.

                  If its still in OPENING state in meta, try to change its state to OPEN. Do it for one of the region and see if that brings down the RIT count to 133.

                  Translate
                  MrNicen
                  MrNicenAuthor
                  Silver ⭐️⭐️⭐️
                  May 15, 2024

                  The current stat in hbase:meta is OPEN not OPENING. I can try setting it with HBCK2 to OPEN but i don't think this will change anything.

                  Translate
                    rki_
                    Gold ⭐️
                    rki_
                    Gold ⭐️
                    May 15, 2024

                    If the current stat in hbase:meta is OPEN, I won't suggest performing any other action to change the state in meta. I suspect there are some procedures running in the backend trying to assign the region again. Do you see any procedure as such in the "Procedure & locks" section under the Hbase Master Web UI?

                    Translate
                    MrNicen
                    MrNicenAuthor
                    Silver ⭐️⭐️⭐️
                    May 16, 2024

                    There are currently no locks/procedures.

                    Translate
                      Terms & ConditionsAccessibility statement