About MrNicen

MrNicen · ‎12-23-2024

We're still facing issues with RIT, and now there's an additional frustrating problem: several tables are not coming online. In the HBase meta table, the state is set to "\x08\x02", which indicates the table is "DISABLING" The region state is CLOSED, but modifying these states hasn't resolved the issue. I've tried addressing this by using put statements and the hbck2 tool, but the problem persists. The current HBase version is 2.0.2.

MrNicen · ‎12-19-2024

We still have 99 RIT's, how can we delete tables that are stuck?

MrNicen · ‎05-16-2024

There are currently no locks/procedures.

MrNicen · ‎05-15-2024

The current stat in hbase:meta is OPEN not OPENING. I can try setting it with HBCK2 to OPEN but i don't think this will change anything.

MrNicen · ‎05-10-2024

We tried decommissioning both busiest region servers, which had combined 80% of all regions. Decommsisioning fails because of the same Stuck In Transition. Luckily the side effect of decommissioning is all other regions did move to the remaining 12 servers, which makes that these two servers now have combined 3% of all regions. To our customers we have a much more balanced and therefor snappy cluster now. But as long as we have these Stuck In Transition, autobalancing will not happen, and at some point will have a imbalance again. It seems that the cause is that these 134 regions remain in OPENING, and should become OPEN, which is not going to happen. What is the best approach to mitigate this? Should we use hbck2 to set the region state manually to OPEN?

MrNicen · ‎05-07-2024

More long ones: 2024-05-06 22:17:11,944 INFO [PEWorker-14] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=65, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_38>, region=d26d9da451ba6ee5d917d0e135f5a7bf}, {pid=66, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_2>, region=ac46967db9dfca5c91369164ac1c9469}, {pid=67, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=8ea10f9305a566d8efadbfcb2c6d87de}, {pid=68, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_37>, region=c038bc8eadb331b104131184d88316ce}, {pid=69, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=d2df8e2457a319244cfa85d898773f53}, {pid=70, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=c32fbee54febb51b1278c139729c9991}, {pid=71, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_61>, region=85a1c4f8c9303a693d2b0baa3f04e576}, {pid=72, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_43>, region=cf3a8c84ee842791b876765071465d98}, {pid=73, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_26>, region=0bb135123341acc459bc2b0e095c571d}, {pid=74, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_61>, region=60936d1899edbaae0788c2d928e34be0}, {pid=75, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=c3763274a6c84f17cafbb831c1a81132}, {pid=76, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=9f4bb127925ef66b2e2f181ee0b69d02}, {pid=77, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_14>, region=846e8ddaec0fffcb97d44480c7cc8798}, {pid=78, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_63>, region=cb50cf460205d12c5bee0e2fb1f4c21c}, {pid=79, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_73>, region=09b701e10fd060bf0901ff8ce55778e4}, {pid=80, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=a228e68c6d231c7d37a15c1c048bef58}, {pid=81, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_17>, region=7ba59a23baf79edc23ebbc5b152936a3}, {pid=82, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_6>, region=e94e891f60a0a0c194700f73504ab4a1}, {pid=83, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_54>, region=be8f1361b364a21f20300aee3c174464}, {pid=84, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_43>, region=1ee1d509aa4c312c6d48e2bcce1d14a2}, {pid=85, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=1621fa407abeeb4c112638af74553342}, {pid=86, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=530c9d659a0416d0f442a09e8070d106}, {pid=87, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=bd644295d91e525001cb1c08f82c06cb}, {pid=88, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_46>, region=69c55317c0eaec7c118fa09308c34997}, {pid=89, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_41>, region=a882652e5d221f3fc56efc5214970553}, {pid=90, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_25>, region=cf45f56e80d4b860220db6589e55fa4b}, {pid=91, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=f6f18f4f98aae325d3aab52560128a2c}, {pid=92, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=bc4a3997c5f8dd993dd3079044bb9443}, {pid=93, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_51>, region=ab6f3e8d44332b4e78a0aae110b26603}, {pid=94, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_29>, region=a9ed6a68f9edc2c93bca66f146d24e85}, {pid=95, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_14>, region=358157917db43e1aab7c64bde66e11b8}, {pid=96, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_26>, region=1491a0474a21f04733065b84a7ea0690}, {pid=97, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_42>, region=143bd5d9cd709b8b346bb166bbce71c2}, {pid=98, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_44>, region=e5eba35899c2e90162b869d8e75a0c76}] 2024-05-06 22:17:11,950 INFO [PEWorker-6] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=99, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_27>, region=fd8364d6adfd11711610317f468a518a}, {pid=100, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_21>, region=81be87773b6c3ffacc0a07c9da8d7dd4}, {pid=101, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=ec8d8cf7f7aaa6e9740a1c43a2942ff7}, {pid=102, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_39>, region=31434aa06033a0edf3cb394a7d64a15d}, {pid=103, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_46>, region=b78b117f1426c5ce0e528eb51ad3ec07}, {pid=104, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_9>, region=fe81d1f92167a6099eb5516668314e3a}, {pid=105, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_62>, region=7b1b806d6b9fcfe5ac2f1d54a7a6229a}, {pid=106, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_1>, region=5b4d78b09c99ac59d10c83fff9303cd0}, {pid=107, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_6>, region=40c174d887cbfab7da2ba1977a60fe94}, {pid=108, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_26>, region=f1bf3decd899563212c071e24508b97d}, {pid=109, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_timespan_index, region=8f794f8b6ac4f70770d0f80a5e9b02ce}, {pid=110, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=3609823c9e0c2bc9c7d65d716e290bb7}, {pid=111, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_23>, region=d004e125f9dfe02fbfc8df621d4c9878}, {pid=112, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_36>, region=a724e0984e13a5dc9160a68ab72f5e29}, {pid=113, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=ae445ff6e5c92bed212429c28bf2795d}, {pid=114, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_31>, region=5c169953e27e09c477c78bfefe9ed88a}, {pid=115, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_39>, region=5c512440cca3566741e6d53e3bb29eb1}, {pid=116, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_27>, region=cab78895641204b76abb7229418ac4d2}, {pid=117, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_reference, region=dd6daf6ff096794f8ebddc0d28049b6e}, {pid=118, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=36add998cd3ce2524203ac9d69906530}, {pid=119, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=587fec8406b9994f1981094ef7e7ebda}, {pid=120, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_54>, region=b92a93a355d6a1ad7ff11bf3e34fb9ee}, {pid=121, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_71>, region=ccc28d5406e44135221ce70fa580cb6b}, {pid=122, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_18>, region=f315381dc89c081c0ff999106ebd0f9e}, {pid=123, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_7>, region=3f3e045bdedf9708c9167aef64477bca}, {pid=124, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=6898c984f4ef6345ff9656f783d0decf}, {pid=125, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=85a72fc500b1d803a1c20e7dd312994a}, {pid=126, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=85df5947445cfa615d210549d4113544}, {pid=127, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=08a4d55e7f60d570e90e08bca932e274}, {pid=128, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=5e6e5a2254aaaa9cd6bcc07787bd7b68}, {pid=129, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=1ff9f8b3fbdd559236bfbf3341a881db}, {pid=130, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=ad8cd76e0a891016efa454b052a4d709}, {pid=131, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=4666ecb9e11eeb2a1a92a5db20e9fc77}, {pid=132, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5a47edc1544c02826a522a8806af7097}, {pid=133, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:wlt, region=5eb06b8067fdbccef433aa61685e2bf1}, {pid=134, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=358a62174fa70ce3e594ac48b65265da}, {pid=135, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_52>, region=a7c63347b3eb5dde263cbfdd65b77a69}, {pid=136, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=f9f52135ba20e1fe76cf371acab6bd00}, {pid=137, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=29431818e46baa2963d67e2635ee83dc}, {pid=138, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_27>:message-bus, region=7da31638b73cf866e7995b1c77cfb2a4}, {pid=139, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_46>, region=0ff03da8b9c98538bbe04d1512144c76}, {pid=140, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=275807cbabad3122b0baddbc1d7352f7}, {pid=141, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_57>, region=887d0888424b0e60060a9cc8659bd35f}, {pid=142, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=987b0dc7408d25aa411924236432d746}, {pid=143, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=f14a6fb3f90f81ee77388964344c2eba}, {pid=144, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=6e505ffbb8a41fd3ec5ffa5678063228}, {pid=145, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_60>, region=14ec3ec9cfd01b8172aac7c2b6f99a6d}, {pid=146, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_59>, region=9b9897f6f40f973796471ab12a7ba9ec}] 2024-05-06 22:17:11,954 INFO [PEWorker-10] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=147, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=00d20f4dd79bcebe1ecfc93b370d1941}, {pid=148, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=b3f91f5c8ef858e6d15c5bb8f7a7cd26}, {pid=149, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:cds, region=ac189d86bf78b4b9a24c476663874eb7}, {pid=150, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe, region=606e3055bcdd795113d6fa794e1d9843}, {pid=151, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_13>, region=2fe2e69dc40e61d76960e2fb9d87731f}, {pid=152, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_46>, region=c78535817a6d3956701a99e89baf6ca7}, {pid=153, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=0cb990ba90c53422b422e6a2b172d4ff}, {pid=154, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_19>, region=1c5466966c1aa44b2191db632806e4d5}, {pid=155, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_10>, region=71c761063de6a50e16750ab15921dabd}, {pid=156, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=5234265075f572118fd896d57b27018c}, {pid=157, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:hgg_client_timeline, region=753d0f68de8e47864ecdf59bc6c3f084}, {pid=158, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_persistent, region=ebc1328fab65efd0d9314130ceac2b94}, {pid=159, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_by_query, region=8f4c816f5ad863cd7176defab9d8b979}, {pid=160, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:dzs_by_query, region=272a0150d5fb7edb81e1230a5d38db18}, {pid=161, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_46>, region=4dd22258f7282411a06566357e44e335}, {pid=162, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_43>, region=5344ff13fb3db964fe346099ccbb5ad4}, {pid=163, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_40>, region=e762e13ed49b02ae79ced188bb68c6cb}, {pid=164, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=4f8445c40820a61633833e2e028d7441}, {pid=165, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_28>, region=c1ed331e961eeeb2bb6fa9d50f73a040}, {pid=166, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=0e3f779f3590717ee562fc40ac26faf2}, {pid=167, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0133fc91fc6d7bef529682456d19a774}, {pid=168, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=9755e4208ced1b938c53017ed30b6c12}, {pid=169, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=cfafb084581d983eef636cd1c09be90d}, {pid=170, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_46>, region=09af2afbe817abcf95dc0cb8825ed9fd}, {pid=171, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_by_query, region=6ff424adf916a4bdfbc4c58ad897d3fc}, {pid=172, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_by_query, region=9c4c92c7246ca209d7a02dca9033f544}, {pid=173, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_27>, region=432de3f682b58a0745464c9335be837e}, {pid=174, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=334b11e67ce7f61b8165e14ad470c9cf}, {pid=175, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_1>, region=9e7c8d1219f6b4f094f5d372aee740c3}, {pid=176, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=aec330cf8c9899290b1cca3d8dbaf7f5}, {pid=177, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=ccd9e06e5f8fcfe9dde1b65133fe8839}, {pid=178, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=9c1eec8741efe2d89c0567484cd638f0}, {pid=179, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=1bbc996cfa734e0fc6c70bacfa5aa4da}, {pid=180, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=7b9fa4da04f481d3258d7fd57b72a354}, {pid=181, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=8e894c6a576efb759fba7cc4da6be1df}, {pid=182, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5cd6925ee92449240ce14a37b275330f}, {pid=183, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_37>, region=48231aab340dff0874a068d75e53fa5e}, {pid=184, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_26>, region=3608dbe215817b2047c73be91604bfdf}, {pid=185, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_by_query, region=5c874fc02cd78b2a2da75bb156789c4e}, {pid=186, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_by_query, region=47211257f5f82a304ffaf05a8942858a}, {pid=187, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_48>, region=b0b022264b64be0b9fe4dc849e1cf1b5}, {pid=188, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_3>, region=01f9fd1bb65e07860fa74ed03266ac9c}, {pid=189, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_quantifier_index, region=2c017dc1457c9390388cd5644d3e3569}, {pid=190, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=bb88b7509375a9e1caa80b2c75d42a6b}, {pid=191, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=7a1d6e6b9390f9be391a2aa246282557}, {pid=192, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=35ae73ec75648c6943cffa7fbfcf42f4}, {pid=193, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_33>, region=adf0fc266fef938f1fb10efb21580df9}, {pid=194, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=f19bade589507b3931911a7a99c2f264}, {pid=195, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_13>, region=c53bd3f2a33d92f039ba4bb16efa518c}, {pid=196, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_58>, region=f3ceda41c170dc8ea5d6b2ef1a057b62}, {pid=197, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=769229ad8d8e8b8498974d2a643ea10b}, {pid=198, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=58ea387d639ee0f764ba90310584c9d4}, {pid=199, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_54>, region=23e746b75b557c89f5f3bb3f0d0de4b2}, {pid=200, ppid=6, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_11>, region=48e2a9ad25715bf409c0ab8390f92144}] 2024-05-06 22:17:11,966 INFO [PEWorker-13] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=201, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_15>, region=b5c4d477bba1dfbdcc2165c523bd7395}, {pid=202, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat, region=bbbc7f79a39a1c8a417a0a7f9054d1a7}, {pid=203, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_60>, region=09cdd7faf2a89a737d813d648e1b028f}, {pid=204, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_62>, region=18a0f41c5988a80a493e05cae42ddf0c}, {pid=205, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_25>, region=bcbdc1f2e315acd61ec815d69df8881a}, {pid=206, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=7ccd5f985123343cd540a4cbdc9a704d}, {pid=207, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_7>, region=b6031e76568526723101704dd52bfc3e}, {pid=208, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_9>, region=c54357dec4f34c820a50ca53852508f5}, {pid=209, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=d53818a4613c3655c8809b488dd13fe5}, {pid=210, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_63>, region=1c56709a7ef1202b2ca3745e178c1ab3}, {pid=211, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_71>, region=4cc6da9099a4fb0cb217a1dba51ef6bd}, {pid=212, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_37>, region=a7f97c7aabc91f42af675c73f03b3546}, {pid=213, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_35>, region=463c29108637a39f2429745d39ff73e7}, {pid=214, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=25ac0b75eae580989551c7d8b3773732}, {pid=215, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=3bc0c1a61c45f68dcbd0af9fb496e7b4}, {pid=216, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_18>, region=e1a8fb3db5e9daa5e2f32d10a29046ca}, {pid=217, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_46>, region=b912321cbb0dcea3280a6015465d5f78}, {pid=218, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_21>, region=87cf47f804e40b1a52a77f5f39b3f184}, {pid=219, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=e17d2f7c9838590707239cc949ec0eeb}, {pid=220, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=4699b0b7032fde3b1736d5d445091add}, {pid=221, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=9fe9190ef7db10a90c2dcf62187eb737}, {pid=222, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_55>, region=f17c894285999f568dc2d13307c30f32}, {pid=223, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_50>, region=59b8a67b8af82dfbb08afa40e6245ec8}, {pid=224, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=be9a5a341a519c132b4daddbd9cd5804}, {pid=225, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=88fa6bfdc0ef4c45b1fe5eb22de6e6aa}, {pid=226, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=d4b53b21efe1bf9a4632dced64ee0d1a}, {pid=227, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=9abbedce0ecb7e64aaab441dfb867922}, {pid=228, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=7137884e41ce4bd388567ccd7c91ad9e}, {pid=229, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=b03a1ac196a2758819ab14bcc2d182b6}, {pid=230, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=abc3172931d7f48c6613a6cf8268ed5b}, {pid=231, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=265d2fedcc86f4101420ff639be21383}, {pid=232, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_46>, region=15d526f60ea517e2129bc373e63ddf11}, {pid=233, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_72>, region=4a074ba4dca8703f0a7ddd87fcd50bfe}, {pid=234, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_47>, region=f494d0b158b7c3dd986314e242d93770}, {pid=235, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_32>, region=3301fea1f596540b4c605a8afb2a83a6}, {pid=236, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_40>, region=d3160c06c71d5ec702d391222f21540a}, {pid=237, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_73>, region=f5085d50d9508c1b11a3e0ed7e98c007}, {pid=238, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_50>, region=933660db79d8c4c42d9cc864493b7d5b}, {pid=239, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_72>, region=714388e2805cc386d7c48850b29e89a8}, {pid=240, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_16>, region=bef909fc3b5fe93a466007e73f8f79ed}, {pid=241, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_33>, region=f445d91aea86819a2fb4278d5448e178}, {pid=242, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_persistent, region=6d22694b0c619bbdd0b8ffe7cfa2ee73}, {pid=243, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_persistent, region=27729d98bbeaa6169d23642af41bb007}, {pid=244, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.app_flow, region=5e934d2af0695112564acfc62766a171}, {pid=245, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_22>, region=b2e955a6ea739dfe1ad4880b40e12d1d}, {pid=246, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=a9339217627c55dd9e2fcc57f9e70fbd}, {pid=247, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=512394fd78cc3fd00e72796f5e468f82}, {pid=248, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=0b90a930e8da2c15cf3765574a0ac85a}, {pid=249, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_12>, region=a6fa48fac62e74f6ce14d81ac2a916d8}, {pid=250, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=6783fc26c108ac5d7e4c59c68d050ef7}, {pid=251, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_60>, region=df6752fb754f6a12769c9f8f6868f5fa}, {pid=252, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_42>, region=c7e68ad2b83cf444a3fe55c597abef1c}, {pid=253, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_9>, region=55ca2633ba2ddee3b6efa7e1b477c60d}, {pid=254, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys_locations, region=17ad83a7769cc356132351b13de1d212}, {pid=255, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_61>, region=04761c39e35764683291abfed1bb32d8}, {pid=256, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_22>, region=48750756b3a6e4414384853246bc646d}, {pid=257, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_66>, region=c8be351bcf2f0d0d119c560d1bc46f1c}, {pid=258, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_17>, region=b8f04ac037145bdf82440c6fde20d8da}, {pid=259, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_23>, region=3dbac8b33c33379e1812b12b6a148d32}, {pid=260, ppid=4, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_2>, region=9ca449153035cf901abd8402d29bf318}] 2024-05-06 22:17:11,990 INFO [PEWorker-16] procedure.MasterProcedureScheduler: Took xlock for pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03 2024-05-06 22:17:11,993 INFO [PEWorker-9] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=448, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl, region=52232b406176da8242980a3735e7b551}] 2024-05-06 22:17:11,991 INFO [PEWorker-1] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=261, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_55>, region=9cabdbf8353308c257cf64514c0bfe69}, {pid=262, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_58>, region=fb87ff575b36c5997d8713378e47f89e}, {pid=263, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=d230357193249bbb002f0785f4c7271a}, {pid=264, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=00af8a539d6212c37c174562639d25b0}, {pid=265, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_63>, region=dba7407a60581148a46c4a135932f335}, {pid=266, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:cin, region=4ee39571768335fe74c62626ef424ec5}, {pid=267, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ztw, region=fa12c00dd61aab37437a09f92f9ef8f0}, {pid=268, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_28>, region=d8dabdd9f1b4080679de2e5cf24b53ee}, {pid=269, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=e7eb7c9c11ef47b3b3d77c0aebc6e978}, {pid=270, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=9672799e1d65bc522c12e4ae8443b412}, {pid=271, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=daaa022ec47e538c1bbdaedebbeefdbb}, {pid=272, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_1>, region=b03bd74027c4b06b0dc80451e5864952}, {pid=273, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=f567546ebf7fa1af2c3c1d913a6cec95}, {pid=274, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=ba777b929557fc60d1730984f6faaad4}, {pid=275, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=0692c4f2914cd80dfe661e3c6ba8ab27}, {pid=276, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_68>, region=2119a81b0ae71aec5dfe6a8635454016}, {pid=277, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_56>, region=f37c3c70820e5dd95166c6e0df6c266d}, {pid=278, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_41>, region=de842bc4866c150a1d0e02cf90743bdb}, {pid=279, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_41>, region=6a8808705fb01b06a9719d0263d4e119}, {pid=280, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_28>, region=4efcb1aa3e0f2cda75f0869656b69489}, {pid=281, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_34>, region=22be35f2c3c4d6024565c71803750012}, {pid=282, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=7dc8f2ceaef8ae01d52da98d3381be71}, {pid=283, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_41>, region=c190dcc9ab4138c6bdfcf4d2050954f4}, {pid=284, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_72>, region=4291fd7de0724af0dac6b5390f01b832}, {pid=285, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=eda5d4393ccb8a30964e057cf1585638}, {pid=286, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_28>, region=5a652c833612877e99cd17a18d16e25c}, {pid=287, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_71>, region=af5b7441e4dcc6be7a729a010f640b98}, {pid=288, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=878f434511529251fb80bd0fa615397a}, {pid=289, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=752a7e93a236de68fd172255e8dc59b8}, {pid=290, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=98aa87b3d0b14083769c178732e65521}, {pid=291, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_23>, region=fccde6c1bde39b9f50756232422bc353}, {pid=292, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_3>, region=10ecf35cb2bbe68c17f7bfc232e6ac9e}, {pid=293, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=81aa44df8d1be4ee7aa7b4b21506cb6d}, {pid=294, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_12>, region=ebb3848ed489a7d12e74033fae54269d}, {pid=295, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao_persistent, region=715fc0851a4a80f9570d15f9f42eeaed}, {pid=296, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_55>, region=c3466fc1a892f1cd76f93d31276b55b1}, {pid=297, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_19>, region=b8a6f89dd367215c3661ea2ad312eb2e}, {pid=298, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=25e33c575daa89c42bc891bc3133a438}, {pid=299, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_47>, region=da4740e4384c819e8b4475aafa668d23}, {pid=300, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_21>, region=5acae61fe9dc5ffb5d6d72e4fefe9062}, {pid=301, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_16>, region=16e4f4914e1b1df0ba72f5853fa525d9}, {pid=302, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=auth:account_activity, region=ed108e6facb210154c031fc595d2986f}, {pid=303, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=5fb1730f3bdbac7016cf22b70b5ebb27}, {pid=304, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=f8fca7a15afbc527020fad926c0daead}, {pid=305, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_37>, region=cc19878cdcfadaab0aa41bccffd1bd65}, {pid=306, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_26>, region=0c0437c6e51070e2a1eb57386c3f2bb0}, {pid=307, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_1>, region=7123221617b10eb342325353513d7455}, {pid=308, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_12>, region=86ccef773c3fd228453a37c170aece84}, {pid=309, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt_reference, region=d3534603c564eda83632939a3fa1224c}, {pid=310, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_27>, region=f151a2ac50ca31cfcc64b46f6859fcd5}, {pid=311, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_27>, region=5cfb990ab8940b24cf7f007e7ef0bf7b}, {pid=312, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=5306405ccd189c1ef2d5962da914349d}, {pid=313, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=90ce8e8e81f751303cae2a5772c21795}, {pid=314, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=89a247d379f35e764b0b22504b78f325}, {pid=315, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_48>, region=bdeca0a3f7f9e5b7c9937e6e8a1d1be4}, {pid=316, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=7e3532e48f693d8d289d813148d6a4f5}, {pid=317, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_56>, region=4ab6f8e662514eba09398bcc767febaa}, {pid=318, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_28>, region=a29c8fc117e40eac93e0f9e119553614}, {pid=319, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=3da669b0bdc76c87363281b8e99dfa8d}, {pid=320, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=e57f5bde53018a69c7c0022493fcd354}, {pid=321, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_41>, region=1192336a35ab7cb2fd8f575cec43fff0}, {pid=322, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_10>, region=278fbd8a05b5d09cb4cb4b409c864584}, {pid=323, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_60>, region=be429de23e05dd59f824f0e12ed4c4d0}, {pid=324, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_25>, region=b7cdce4d0b956eaa69a766672b3090a9}, {pid=325, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=bc8adad0a53341adf39cce1f5bcf3e9d}, {pid=326, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_25>, region=89b61005b4de954cf1bd8a1ed046c294}, {pid=327, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_3>, region=c2890cd6fe28924884b902c60f6ff434}, {pid=328, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_3>, region=d43cb6cb67cb7ae0ddb3a577bfedb707}, {pid=329, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_54>, region=348826fe55734025419e00d13f178218}, {pid=330, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_60>, region=76a52c147a6b738fee00abf0a8d38cfe}, {pid=331, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0de76b556fbaa3bd1a3aa02dcb9c3a3f}, {pid=332, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0238767c59208b1368ec21218f7361ac}, {pid=333, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_49>, region=707c27be94d45223eaeb876b0929793f}, {pid=334, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=bea7ac8a617a7380cd5c2545e420d3ba}, {pid=335, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=c51df52ff432403504fcb9c5b650290c}, {pid=336, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_40>, region=d94808418d2327e2e61a3d9cd141cc5a}, {pid=337, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_11>, region=fe4ce65f39cffa3fa363e07ad51a94b2}, {pid=338, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_5>, region=1a705b0e8eeb0d1f9561ea34a7d2f7f8}, {pid=339, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=4aa2b82b3718192d0c0126855ff9cb05}, {pid=340, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_46>, region=c3e304de319ea07e830463cb8212bcca}, {pid=341, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_57>, region=0a179c22a5ce6f8c93fc3cefe4a1f864}, {pid=342, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_60>, region=0b3ef2bc9d82cc296fe41b46de26ed78}, {pid=343, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_7>, region=0deca40e0bf3442b4c73726b16e19743}, {pid=344, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_12>, region=b541f926df73acfada2e2a35642d7c07}, {pid=345, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_73>, region=aff63924d0be9e25291bdba5d2bcaab1}, {pid=346, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_15>, region=691aab81768ff6a808b605f7532cbd1c}, {pid=347, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_35>, region=8cf9e7c507961c6b6cfd5e09397e274f}, {pid=348, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_34>, region=1e1ab991cd7fc4fcea21491fc4271e50}, {pid=349, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_43>, region=3deadc750ca47a175e7bef12b9396233}, {pid=350, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_55>, region=8bc319cfb1232f5e9d8cdab58fdb73f5}, {pid=351, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_1>, region=ff435f619d4ebca70dccea01c1544419}, {pid=352, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_32>, region=5ee934b10ba0d0a3ad5187e8fc4ebc6a}, {pid=353, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=28e539bfa8ad97ccdd163eccbf272a29}, {pid=354, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_22>, region=44bb488a9e65a184488146ffff903e65}, {pid=355, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=dc9d9c2f14473333d6d05754356d9ac6}, {pid=356, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=1f3a3259f905d22af35549c2db081544}, {pid=357, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_11>, region=c8dc2b7ba47fdd1b075dca81cebdb645}, {pid=358, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_29>, region=510e50899e5fb5e8ccfb71ff63c5638f}, {pid=359, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=989ca9c8a3f5a1756ccdc1089fcc4cc1}, {pid=360, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_6>, region=e8937927d7af6ee8273fdae233beedaf}, {pid=361, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_13>, region=93c56196d1938a5f4136567602317244}, {pid=362, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=8986a928fe5c5e4325198664ce5d17fa}, {pid=363, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:cla, region=8402fe5144954dc56d45adadf67b9cbf}, {pid=364, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=d70ca2c40b9ffc76d58e7a530789a8c0}, {pid=365, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=138cbafeceebde3459edf79fda73d450}, {pid=366, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=cc58fc9fedae3fc75c8655986902ba6c}, {pid=367, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=479d22ee3490e4191ef1d43c17fda93b}, {pid=368, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=5febafd6d035f8e20dcde9a4f74e61ce}, {pid=369, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=b162fa7207b2a21cd616522946bfab3e}, {pid=370, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=4f968e6ede745833d6c639d0a9ff2807}, {pid=371, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=9e4c23ea1ce446781c57a72d59ff75a1}, {pid=372, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=f4e41961f3920e038f0a5aec72a7e667}, {pid=373, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=d382dff4e9eab83e22dfd63943168677}, {pid=374, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:wlt, region=27021a1dd9b3a3cf0ac8d9022746150e}, {pid=375, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zao, region=4f3f5a5bf17be1aa79a03e94c1285576}, {pid=376, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zbl, region=9a948d0c445d3d810f402b9cf77f8a85}, {pid=377, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=bea13c1ef47833992db43210e38ba678}, {pid=378, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_61>, region=816769a8ff9c8d34a484441ef39c2b1a}, {pid=379, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_15>, region=f2d1d8517f4541a4916db74e63e9df0f}, {pid=380, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_70>, region=c6efd6b6836e968552d8f531ff1358c9}, {pid=381, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_25>, region=2e74a82a4b3f6297076ef6e25261b460}, {pid=382, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_7>, region=56666ded0297dd0136797f4844b33a0d}, {pid=383, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=eed5ba2cf6b68aa4ff6149759223c266}, {pid=384, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_63>, region=e04ad9181d3577dac7c126c66c721650}, {pid=385, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_15>, region=8bc9b9261ecb41b744ae8fcabae8bdfc}, {pid=386, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_65>, region=ee223a2d3c84bd2a35227124ccd3efb8}, {pid=387, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_8>, region=8f388296a24cf746ac8762a16698ca22}, {pid=388, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_58>, region=f86e25e7ca220db5c908669b3de1b1f3}, {pid=389, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_42>, region=05219a664defb595ca7658a1bd7d511a}, {pid=390, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_34>, region=ee0bc298e2fc3b8dd0a15fec2ae894d6}, {pid=391, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_23>, region=96eff57175c090f342530b2f41b2de99}, {pid=392, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_23>, region=c8020dff04475030b1ca6d9fb66475e0}, {pid=393, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_62>, region=9fcc9a48a801657d7abea81a6f41b3b4}, {pid=394, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_9>, region=5e39e64343d157bca285adf8cd1b4753}, {pid=395, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_12>, region=7695f48743f8d50e36b783ba93ce5bd7}, {pid=396, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_53>, region=4265434f477b4eadec5918f00851ea86}, {pid=397, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_72>, region=91c5fa07f5bd2297ea2b1c3837b69407}, {pid=398, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=3f78b16536cf86845f913b6d333d799b}, {pid=399, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.app_flow, region=662005b2e8162740152cdca4d80ab677}, {pid=400, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_14>, region=3b9fe4b67b5a48ca8e5c6ac5fb7006bf}, {pid=401, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_64>, region=ee909bbd56543c5191161088fd5a806c}, {pid=402, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_63>, region=d6d13581484e69ca42f4702a1a6af9f0}, {pid=403, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_63>, region=5d854f1eb7bcda240d0f6d6bb8545815}, {pid=404, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_41>, region=9d9ccd2bc40a4c358e41a86ee9957c15}, {pid=405, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=00c8319a210d90772fd031a45c88ec36}, {pid=406, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_41>, region=f3a773f2add03d4e976fa8a5ab200c0c}, {pid=407, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_51>, region=3bf60504f9714870997e24ed142d3faa}, {pid=408, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_72>, region=4b2cca0a0cf2723a8488e88bafacf95e}, {pid=409, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:services, region=07466e00c206b9b934417f0040059a67}, {pid=410, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_22>, region=c3d9ee32a04d228846ea0bee462f4e55}, {pid=411, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_26>, region=abc6e8e483eb33ef88e9026e6ba8dfed}, {pid=412, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=73ef17461282d50cff567f089e432bac}, {pid=413, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=8e92a16af113fd860455d91f37aae3b7}, {pid=414, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_11>, region=16fbec1044017e15b5b3b514b17eb59a}, {pid=415, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_6>, region=6ad83e8d695932a4075f6213fc924e07}, {pid=416, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_24>, region=7ba8496b2e2774e476521870581eb137}, {pid=417, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_72>, region=77afca33126ac4a4d3650471e4f51852}, {pid=418, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_4>, region=4b3368505a01a7db9df97335fa2c24b8}, {pid=419, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_31>, region=c942b0795588e8965cfadeb8b409f510}, {pid=420, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_40>, region=15e502c41f2b932092c32a8613f4948e}, {pid=421, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_23>, region=0d51be0e6f91c2df9e50fb6bd888ae5b}, {pid=422, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_43>, region=e6eb36d4c9a022460a70b71cbb85038f}, {pid=423, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_70>, region=fb283affe3bde8f245524fc2a4465e66}, {pid=447, ppid=2, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_13>, region=3129677490b14ed8ce90f0c386f5af60}] 2024-05-06 22:17:11,991 INFO [PEWorker-7] assignment.AssignProcedure: Starting pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null 2024-05-06 22:17:11,990 INFO [PEWorker-8] assignment.AssignProcedure: Starting pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null 2024-05-06 22:17:12,001 INFO [PEWorker-15] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=449, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr, region=2be9537800624651fdc23200b9d052e6}, {pid=450, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt, region=6d268b7ba4e5a74ddb13a99f210b7aa9}] 2024-05-06 22:17:12,038 INFO [PEWorker-4] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=451, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=c61efe7f834bd5e8fb1ef573c6476027}, {pid=452, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=466adde637ee5d7641d5c2105074b9f4}, {pid=453, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_32>, region=3917850b8d9ad87fe38019f012fe2add}, {pid=454, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_41>, region=eae44441062d5dc8c0688b8f055e60a0}, {pid=455, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_41>, region=46707cfcb32eb52dbe9e8c8203afa443}, {pid=456, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_46>, region=92d3626516835796829a9a45c5956c4b}, {pid=457, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_8>, region=b2bb4b7465cdcfebe64237094ce948d2}, {pid=458, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ztw_locations, region=6adeced16672c2f98bc3c1aca6e55db0}, {pid=459, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:hgg, region=9330c212add67cbecd0e4bbb65f72178}, {pid=460, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys, region=7afb999201aeeea87e994597952c197a}, {pid=461, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_11>, region=1f897c1321e379b023db8dcb955005e0}, {pid=462, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_66>, region=e694b978e0c122633fd837284053f61b}, {pid=463, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_63>, region=30a70ae706e91a222274166de838111a}, {pid=464, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_63>, region=83a74451025580bb7f9d0cd11d10a139}, {pid=465, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=6d3586ba82d879a8041a830e463cee79}, {pid=466, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=1c46655faa39c4ab57339fae984cfbc3}, {pid=467, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_61>, region=53e0297c25320786bb23231acb2b6b04}, {pid=468, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=cacc72e20e62f4067f47519ad1d267ba}, {pid=469, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=a7c7ae317038d7a576618d3fcd27812f}, {pid=470, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_63>, region=b8100430c8dce21a0d29651e1cdfa327}, {pid=471, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_66>, region=a94e776a27155a45f4ae995a595598df}, {pid=472, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_19>, region=f708a1b93798a3a32f57ac88298f44f3}, {pid=473, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_7>, region=8680fb202f5002291d01e33f6ae0c677}, {pid=474, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_64>, region=cb2f8e4706d0671b296cbfa6dd64d80f}, {pid=475, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_39>, region=827abc98319f1cb36091de23bb8d0c4a}, {pid=476, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=e7027e77bc5a3502b93817fe70f5b1a0}, {pid=477, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=f84659ec3a9df6d12f266e23d77dfdbb}, {pid=478, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zys_quantifier_index, region=9585cb481dab76c8289ae37a2070e733}, {pid=479, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_2>, region=73ad4e436fe8a4c454ad8cebbb5229dc}, {pid=480, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_3>:<TABLE_5>, region=a5464cc09a771f2bf2eac2d724cc2fea}, {pid=481, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=c1e760cf795e36d5c930114b7a3b6c69}, {pid=482, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_63>, region=83f8632fc23f1acbf2c1e9766a75b782}, {pid=483, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_66>, region=cc3f02439ba1b94dad139118a39e7fc4}, {pid=484, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.entity, region=559e67f55cbb06b0130690c943b809ca}, {pid=485, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_30>, region=913134de6319bd6745c9b89c4fc187dd}, {pid=486, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_63>, region=fca99618699b16e3cbecc721c923f547}, {pid=487, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_66>, region=97ec5bc08b8ee3e9b0465f34aaa1b686}, {pid=488, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=e16719230d3f0e3f93ea2812c58b5be1}, {pid=489, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=ce4aba20a0fc64b3a5cb7875bcca9bb0}, {pid=490, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_17>:<TABLE_46>, region=8b87f38f2951c4ea3c61d07277b7b896}, {pid=491, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_29>, region=174010da7a14cb23ed2d4cda5066f614}, {pid=492, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_1>, region=abbe0ec0457aab7b4153e6f2fd06fa85}, {pid=493, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_39>, region=1116fa6dd513258f3d5affa052962bfa}, {pid=494, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_38>, region=bb186abfb55ebdcc607b27d71668e200}, {pid=495, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_66>, region=e837a6cdeb7ef4c0145ed521200a52e2}, {pid=496, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_67>, region=7c97ccf6990a6eaea378421b8c577e3d}, {pid=497, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_18>, region=4f857594b08360c5522bc4a3d4097997}, {pid=498, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_12>, region=c6bdd90b36bd98a20781219b6af2d303}, {pid=499, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_65>, region=ef6516fb5fe98a69df64db8724b3517b}, {pid=500, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_4>, region=9a8fb840b2caadcb1c179474eaa635b1}, {pid=501, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_42>, region=0a1b3f8f3d99e378328245963da407cc}, {pid=502, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_timespan_index, region=4bbf2884d5af0d06f627ad9d193837d0}, {pid=503, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_26>, region=35c34d62af1e731f278c45961a307836}, {pid=504, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_23>, region=abf3962c01cd35820afc1399f4d6a405}, {pid=505, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_18>, region=ff1798effe6fc9d1f37cedf02e58e798}, {pid=506, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_61>, region=bf08e209b6e07121851d274a628fb856}, {pid=507, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_38>, region=f812525ecdd82c440b8dd8d5e958e2ec}, {pid=508, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_50>, region=e8e3c6220bef865656c4a6d568ca8008}, {pid=509, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=fb8af02c18de3e29eb27f5f1721e4f7a}, {pid=510, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_72>, region=9ee442a3b9ae69e207bcc6411cb72c58}, {pid=511, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_42>, region=480f99176d21b082375a33fe3bf12d89}, {pid=512, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.subapplication, region=702cd74f208b5cffd27d45b0177634ba}, {pid=513, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_11>, region=8335322953acc00d697149fa02bc039f}, {pid=514, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_27>:<TABLE_69>, region=66136ae9052fa89bef76c42bef64c009}, {pid=515, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_24>, region=399e943e9954806aaef8d6a2bc70e1fe}, {pid=516, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=59e65b38b13dba3ea251a733ff328e59}, {pid=517, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_59>, region=ca8753bebe73cc6f6d5c664d619c2ae3}, {pid=518, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_27>, region=767032840c65e8f9cd5089a22ee1643b}, {pid=519, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=8d5801804805600bea72ce5cd9446c48}, {pid=520, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_55>, region=bb8faf48d39cb9c5667c4f4b81e37388}, {pid=521, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_45>, region=758702557664af51f95e7892184a84c2}, {pid=522, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=b31f6d9cd79b77f7b387a48f1578f283}, {pid=523, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_63>, region=85f527a85b1459ba4d0515cba9f33d4f}, {pid=524, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_49>, region=a501efad6e61a886accd69096a51de0f}, {pid=525, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_55>, region=d685720514e4b88309023b6a502966b0}, {pid=526, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_43>, region=49d9e7be19851aa294cc7cc4b55d9e40}, {pid=527, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zgb_persistent, region=8f72a863e1c3d5720cd3bbe6c7971976}, {pid=528, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_25>, region=25b787ba9ac7be4bfaf16d4d5a48b1c7}, {pid=529, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_72>, region=12cee6e8df27cb239ff357e0cce911f1}, {pid=530, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_26>, region=3667a7af201512c963c36928d0beec2e}, {pid=531, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_9>, region=a4ef2c4778117fd18df53fbe2db96e16}, {pid=532, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_27>, region=3e9a743da8b6729ebcf46c11dd9592c8}, {pid=533, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_29>:<TABLE_42>, region=c0dda7faac9764443d415ace31bd457f}, {pid=534, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_6>, region=658d35c14ffcf80187acf4a7e588d8b5}, {pid=535, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_20>, region=3f8100c740920f820672a13942110ced}, {pid=536, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_53>, region=a8886ffa4fe8011cf3644aaa8686f5cf}, {pid=537, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_60>, region=79c7205c1a713ccb93057d36df90fa4a}, {pid=538, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=588b3fe66bd63bc7b9aec55554e6baf8}, {pid=539, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=5dd7d57f8858f77f59493d767577fbe6}, {pid=540, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=85e30b21a4d068e598b850cf405b5a09}, {pid=541, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_53>, region=8d02e4646ea1e3b269b0df8667c49b5b}, {pid=542, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=24636244d51e67e3a35843d2b44b0e06}, {pid=543, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=2438f52750eed160df63520c76e04bc7}, {pid=544, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_19>, region=09a5df83079bf8b9e2f0ce3e05d35e9e}, {pid=545, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_27>, region=d3d4f6e36922bf18cf888da9e48daa8a}, {pid=546, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_6>, region=4e32e01434dfe469101bb291c037b861}, {pid=547, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_53>, region=9d63646dcf52a57e26314aafda41ac3b}, {pid=548, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_37>, region=ab3395aef65ed976125fe46e5a837d36}, {pid=549, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_71>, region=c32c2e3905359e69ca142395a088cf97}, {pid=550, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_11>, region=3bfbdc17c6f863c691e9795dacb68c4e}, {pid=551, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_34>, region=c1860525b46b5bd2f9586d2c87504820}, {pid=552, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=4358f469f4d74d50d66331ee74277e02}, {pid=553, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_43>, region=9ae4e344d28cf89d9781d9b3cbb2e73b}, {pid=554, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=0e0051bd2e1b2cfa869a623cdc909763}, {pid=555, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=c6b220e50e28a3b3bf4a5f29fab3f2cd}, {pid=556, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=92dfdd8a2532cb399e6e67eba5664a79}, {pid=557, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=0038e7ee32d606dc00bace075e151fc5}, {pid=558, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_24>, region=02bb3db34a8e992383401b41879a8ad6}, {pid=559, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=3e19e390336d24abb392ce5d53081cd1}, {pid=560, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_26>:<TABLE_41>, region=752425315bec32b835fd86bcc9e1d955}, {pid=561, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:cla, region=0e8e45dc138f0aafc891a7611c2b0cb7}, {pid=562, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=376d53328d29de238443148140de46a8}, {pid=563, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=b4c74b65072baf60835a410145b819c1}, {pid=564, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:dzs, region=d1194857a1fa4d9c9c2d57a2e190d5ff}, {pid=565, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=47b0efbd946486e7bc824ec5a9150aac}, {pid=566, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=c85e4a00d06036294f35f263269bda4f}, {pid=567, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=e6c0e6d7bbe6ad01be65e9fe57aba426}, {pid=568, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=cd8e266bde4dc549910a88146e070553}, {pid=569, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:hgg, region=89b70ce278744658a639eddf6d90f44d}, {pid=570, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:idp, region=89ed47dfd54aaacce7298ed7ffa135dd}, {pid=571, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:ijs, region=510c369b6cf0c330a9415854efd36873}, {pid=572, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=4a49c9bac8263915c77b8b116f85e5f2}, {pid=573, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=f315c4987d7f8e96d35b5a5b836c18a0}, {pid=574, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=36546490b285ad7f21af0b19007480f7}, {pid=575, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=00cfea50f773a27134e67aac6dce8fa1}, {pid=576, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=0c4ffaf5aeca347a19cf9eed3ddc5358}, {pid=577, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=509d458e718f97e075e26d10be568e3b}, {pid=578, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=09db0b1c02f96a50d1fff02f1c8af7db}, {pid=579, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=5e800770355a88f427488b629abb57cd}, {pid=580, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=83f7dc30a5f06d5adb13524588855818}, {pid=581, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbb, region=291b502a62b078b5586d649885f5d475}, {pid=582, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:tbe, region=db1e76f15e3d73523ebdaf1c1c06abb2}, {pid=583, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zao, region=38e95d30bba9428234fb15221dc6ce5f}, {pid=584, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_12>, region=9b1f61309b94ada8009a178ddb4e4636}, {pid=585, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_59>, region=8641d6490c9b56c408d94ac096740e3d}, {pid=586, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_46>, region=354f0c8a1370815286b51a4b1582d3f8}, {pid=587, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_14>, region=5b2dcd3b95616450baf72ee664188293}, {pid=588, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_19>:<TABLE_29>, region=6d9ca2a6acfe4ac471cbccdc120c3b04}, {pid=589, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_24>, region=2e2b88b0bfa20453c49028eced20c2ee}, {pid=590, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_7>, region=6a8efda89913a2b556771c59c3047345}, {pid=591, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_27>, region=24c5ee2022b934078c0f78d21deff906}, {pid=592, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_reference, region=dc0291681b786728f6cd21ee6a23bb6f}, {pid=593, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_42>, region=342672c1865f212a3f0fe41dfe989b86}, {pid=594, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_12>:<TABLE_61>, region=a56d1d3443e867c6f2e4d4204556589c}, {pid=595, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_54>, region=dbac96812aba03d24e549e0b89c8650d}, {pid=596, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_35>, region=3d41647bd0a155cfed283088a1179728}, {pid=597, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_34>, region=3415b85569434236da100aaae8fd01ca}, {pid=598, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_30>, region=d8582c530ca270bea5c3fb3bef040de3}, {pid=599, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_25>, region=2490192492c7652d71ddd69df4f8b8c1}, {pid=600, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=5d4fc7dc0f39106a648bcf47f1552680}, {pid=601, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=5d76b5dcd749c89e2ea2977711a06c7d}, {pid=602, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_persistent, region=6595469e6bb9dfe0e58100daad516e03}, {pid=603, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_31>, region=9ba2f6b08e1da4bd2d47a4df7a7032aa}, {pid=604, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_57>, region=8fcc91a518f6f34a036b6c5e6ee7f4c9}, {pid=605, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_34>, region=bf077734bf83bf98d2d1f078fa2dfcf5}, {pid=606, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_14>:<TABLE_26>, region=d51433bc454a247671a6fc8d8cc34507}, {pid=607, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_61>, region=b8b53b12a6855eaaf23f2fcd86d80935}, {pid=608, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_65>, region=e6955787e6997e9f09d6e685031955ef}, {pid=609, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_36>:<TABLE_71>, region=5560528e9a2a93d789b7be1fc22a2c16}, {pid=610, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_72>, region=fc009e8566c9b8db9fef4977d10f6cb5}, {pid=611, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_29>, region=788e6997db951c8ea7fb9d637f8487d8}, {pid=612, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_9>:<TABLE_25>, region=ee67ec547647c28fdeed0101b8296681}, {pid=613, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=3e0baf03cc1879ce98f594480ffe4632}, {pid=614, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_59>, region=0f168557d185be24d8236b1f577e7c7b}] 2024-05-06 22:17:12,053 INFO [PEWorker-5] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=669, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=75a9ecc30a2fd3276de9ac01c4a3b6af}, {pid=670, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_41>, region=7e82b7202431c0772d57b96c859c815d}, {pid=671, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zao, region=585a71127b4393c583f2b1a39536009e}, {pid=672, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=b077d9c6af116f153cfbbf156dde77f6}, {pid=673, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_57>, region=ed80d78492d5b36bd676daf7dff5aa26}, {pid=674, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=fc8f9b834ef15ada068f9054f0a3246e}, {pid=675, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:ijs_by_query, region=e14adc8133fa8e50791a7ea74c88b1ee}, {pid=676, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_60>, region=55c76fd6461d42a68ec583d1be33b43a}, {pid=677, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_53>, region=07baa87f9c9c904c72163ec6e2d9adb3}, {pid=678, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_4>:<TABLE_43>, region=15de5b3bdebed284129ddaf35009b932}, {pid=679, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_12>, region=ffe92559774b50614e7551275f6594dd}, {pid=680, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_67>, region=1b814c1cbf8c1e9de7565efd3b25f990}, {pid=681, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:wlt_timespan_index, region=0adb337e4d98c90d5301d9964f4282ea}, {pid=682, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_5>, region=c9826b585da29b6840208c1662c03086}, {pid=683, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=d8a1bea07427eb558c39190f8f946746}, {pid=684, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_46>, region=7def6eddc52abcd19bab85839914a1a1}, {pid=685, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_71>, region=dc099e9ba7d0d3f5f9aa6be10b80028e}, {pid=686, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_41>, region=8db2e5cada5820754b004acbc880ce17}, {pid=687, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=c799db86045094fd0320e5fb37323173}, {pid=688, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_14>, region=0bca31436f8ae42d4d06ef1dffb6b7fe}, {pid=689, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_by_query, region=9c7f4d8ef637f8e2fb0ead64bdcc5c36}, {pid=690, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_by_query, region=fd6b15de65bba3d43436505017e5b2ea}, {pid=691, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=713e83fb0544e3f507c4cfe1b9f510bf}, {pid=692, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbb_persistent, region=0e79e0171eb7ed47389b4eb984fca553}, {pid=693, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_46>, region=09ed5c26faffb87ca8c6843a0a7323ed}, {pid=694, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_1>:<TABLE_8>, region=8e267c07f543e25bc2a320707a204d21}, {pid=695, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_35>:<TABLE_10>, region=a362ff12ae734f09c9dad5e2cff7dddb}, {pid=696, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_63>, region=77127d1dca307dc74ccae9e12a23078e}, {pid=697, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:pat_persistent, region=8b4402597026de84717ff39705f490b5}, {pid=698, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:tbe_persistent, region=43c366fdcd5094a13209c7bae30b7a9c}, {pid=699, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=9469573427d2c76940adac29705267ae}, {pid=700, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=55617f37c3820f05aad9aef85e4d9346}, {pid=701, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:grh, region=16e0ce258462e89603eb7697228ab6f7}, {pid=702, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=ecea80ddf358a89c803ee0f1a8cc8b5f}, {pid=703, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:nbr, region=1c4091e7e7cedafb58beea23cd2959c2}, {pid=704, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:pat, region=50f17b259f03ce2f7bf7ebe0b41c5e45}, {pid=705, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:zys, region=1d3569bdb8ae48721d91d0783e77cd92}, {pid=706, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_48>, region=b2a4ab25b11f009e6607d70bc9ffcf3e}, {pid=707, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_46>, region=dbdbc86496bb9c98e638323bbefeee5c}, {pid=708, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_56>, region=d41f6b1862d2ae11100d85565f8e5fec}, {pid=709, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_37>, region=7436bb293ecaffb269b22bf9af7c8576}, {pid=710, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:nbr_client_timeline, region=d8eaadd1083d3a03848396f2c8d8c8ef}, {pid=711, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=2fbd1ff98a1b2442e718c43ac48f4734}, {pid=712, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_66>, region=e06908bb602a88955af02e853dbe37df}, {pid=713, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_3>:<TABLE_31>, region=177abc45acf52c795b6dd6f67145c5d6}, {pid=714, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_63>, region=9fa16bd1cdcd629bccd2f6ffd381a7e5}, {pid=715, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_34>:<TABLE_28>, region=e94e6814d161dffd12834ae227ffef45}, {pid=716, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_5>:<TABLE_63>, region=ff9ceda3a8b8bd17fc4d627476ba2255}, {pid=717, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_1>, region=301e73eb5950ae0b8b5ba2b97a30ca13}, {pid=718, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_53>, region=099257cfe303f0c20275b4a8e8c3e276}, {pid=719, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_38>:<TABLE_62>, region=8172981d31c57437920ab13f6bea9a4c}, {pid=720, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_13>:<TABLE_57>, region=59425dfd7861755acfcaf4a741a9e402}, {pid=721, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_26>, region=8d9d3eccb0b62fb9746166f35139169d}, {pid=722, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_40>:demo, region=370539314e0bf6e2133520805e16e594}, {pid=723, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_41>, region=fca59829f9914d8bc0ef56065f18386e}, {pid=724, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_21>:<TABLE_51>, region=00e62790dc24813dce28875ce8cd5b3f}, {pid=725, ppid=13, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_40>, region=0d9ed0f5f7fc63838db85dc1c406c54e}] 2024-05-06 22:17:12,097 INFO [PEWorker-11] procedure.MasterProcedureScheduler: Took xlock for pid=19, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:zbl_persistent, region=ba37bf35a4430884d1a6ebbca95a8387 2024-05-06 22:17:12,097 INFO [PEWorker-16] assignment.AssignProcedure: Starting pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03; rit=OFFLINE, location=l224-n1.<DOMAIN_G>,16020,1714828981244; forceNewPlan=false, retain=true target svr=null 2024-05-06 22:17:12,099 INFO [PEWorker-14] procedure.MasterProcedureScheduler: Took xlock for pid=20, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_39>:grh_persistent, region=68531e269838537d534e81683e2a7097 2024-05-06 22:17:12,100 INFO [PEWorker-6] procedure.MasterProcedureScheduler: Took xlock for pid=21, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_28>, region=72f61b803e206cc95dc2a05b1112d39f 2024-05-06 22:17:12,100 INFO [PEWorker-10] procedure.MasterProcedureScheduler: Took xlock for pid=22, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_5>, region=f696c18f229b7b6525405d55afdd57cb Skipping INFO but keeping WARNs here. We have seen these before 2024-05-06 22:17:20,363 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_1>:<TABLE_51>, region=ec4e4dfe2c2bcf372ab8185150d41e1c reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running 2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_4>:<TABLE_48>, region=7e3c25195bd235046fa37428a5a58eb7 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running 2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_5>:<TABLE_3>, region=ed1abdd5978b355bc547f3322bbc79bd reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running 2024-05-06 22:17:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_7>:<TABLE_28>, region=319217e06a1140bbf8d3e569e1931db3 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running ... And more of them: 2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_4>:<TABLE_33>, region=d41e7923f187ee13d081591227c7805c reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running 2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_5>:<TABLE_40>, region=7cc3121f8462d670c938bd3844861436 reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running 2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_8>:<TABLE_12>, region=2788b213d23851dc98611dfc43935e9e reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running 2024-05-06 22:17:20,619 WARN [RpcServer.default.FPBQ.Fifo.handler=35,queue=0,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.<DOMAIN_G>,16020,1715026585040, table=<NAMESPACE_9>:<TABLE_68>, region=460607d7a84d9b03c58f853ecb5a2b73 reported OPEN on server=l225-n2.<DOMAIN_G>,16020,1715026585040 but state has otherwise AND NO procedure is running ... And much more of these: 2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_4>:<TABLE_48>, region=7e3c25195bd235046fa37428a5a58eb7 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running 2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_5>:<TABLE_3>, region=ed1abdd5978b355bc547f3322bbc79bd reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running 2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_7>:<TABLE_28>, region=319217e06a1140bbf8d3e569e1931db3 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running 2024-05-06 22:17:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=44,queue=4,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<DOMAIN_G>,16020,1715026584782, table=<NAMESPACE_7>:<TABLE_52>, region=a989864b76285718567c090494024e34 reported OPEN on server=l230-n2.<DOMAIN_G>,16020,1715026584782 but state has otherwise AND NO procedure is running ...

MrNicen · ‎05-07-2024

The following ones are odd: 2024-05-06 22:17:08,626 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=audit:dzs, region=3da7e18863977749267f48437295889d ... 2024-05-06 22:17:09,760 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_13>:<TABLE_46>, region=0486a586e762a8fc837ad4cf3f48c0a1 ... 2024-05-06 22:17:09,763 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l227-n1.<DOMAIN_G>,16020,1714828981420, table=<NAMESPACE_13>:<TABLE_63>, region=8b12508d2ef09a011875540b383135dc ... 2024-05-06 22:17:09,931 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l228-n1.<DOMAIN_G>,16020,1714828981415, table=<NAMESPACE_17>:<TABLE_46>, region=fcf92fdae49288267f5b94f80bafd186 ... 2024-05-06 22:17:10,046 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l227-n1.<DOMAIN_G>,16020,1714828981420, table=<NAMESPACE_19>:<TABLE_12>, region=7766e1e93ed788e0233d1286968fc132 ... 2024-05-06 22:17:10,046 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_19>:<TABLE_12>, region=969d766f8ecaf8074572a97dd7a4ad89 ... 2024-05-06 22:17:10,047 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=<NAMESPACE_19>:<TABLE_12>, region=2d84f975f555ee262c64618055075ccf ... 2024-05-06 22:17:10,047 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l229-n2.<DOMAIN_G>,16020,1714828981440, table=<NAMESPACE_19>:<TABLE_12>, region=2f1f50ee36b7d30b9f234380a02cd154 2024-05-06 22:17:10,081 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l226-n2.<DOMAIN_G>,16020,1714828981235, table=<NAMESPACE_19>:<TABLE_41>, region=dcf0f06facb516b3a05803f1510d35ea 2024-05-06 22:17:10,082 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l228-n1.<DOMAIN_G>,16020,1714828981415, table=<NAMESPACE_19>:<TABLE_41>, region=ba1f621709cf61c2c7ac523765f11a2d 2024-05-06 22:17:10,214 INFO [Thread-16] assignment.RegionStates: Added to offline, CURRENTLY NEVER CLEARED!!! rit=CLOSED, location=l224-n2.<DOMAIN_G>,16020,1714828981192, table=<NAMESPACE_21>:<TABLE_41>, region=38e9a26a5df320c95349afc7ca374604 These keep repeating 2024-05-06 22:17:09,691 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740 2024-05-06 22:17:09,691 WARN [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628 After assignment.RegionStateStore finishes this first round of assignments in [Thread-16]: 2024-05-06 22:17:11,444 INFO [Thread-16] assignment.AssignmentManager: Number of RegionServers=14 2024-05-06 22:17:11,459 INFO [Thread-16] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1715026691459 2024-05-06 22:17:11,460 INFO [Thread-16] assignment.AssignmentManager: Joined the cluster in 3.0330sec Next, we arrive at messages from master.SplitLogManager 2024-05-06 22:17:11,656 INFO [PEWorker-1] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting is empty dir, no logs to split 2024-05-06 22:17:11,661 INFO [PEWorker-1] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting] for [l229-n1.<DOMAIN_G>,16020,1714828981427] 2024-05-06 22:17:11,663 INFO [PEWorker-4] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting is empty dir, no logs to split 2024-05-06 22:17:11,663 INFO [PEWorker-4] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting] for [l227-n1.<DOMAIN_G>,16020,1714828981420] 2024-05-06 22:17:11,664 INFO [PEWorker-1] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l229-n1.<DOMAIN_G>,16020,1714828981427-splitting] in 3ms 2024-05-06 22:17:11,666 INFO [PEWorker-4] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l227-n1.<DOMAIN_G>,16020,1714828981420-splitting] in 3ms 2024-05-06 22:17:11,667 INFO [PEWorker-16] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting is empty dir, no logs to split 2024-05-06 22:17:11,667 INFO [PEWorker-16] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting] for [l224-n1.<DOMAIN_G>,16020,1714828981244] 2024-05-06 22:17:11,669 INFO [PEWorker-16] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l224-n1.<DOMAIN_G>,16020,1714828981244-splitting] in 2ms 2024-05-06 22:17:11,690 INFO [PEWorker-11] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting is empty dir, no logs to split 2024-05-06 22:17:11,690 INFO [PEWorker-11] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting] for [l226-n1.<DOMAIN_G>,16020,1714829249771] 2024-05-06 22:17:11,693 INFO [PEWorker-11] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l226-n1.<DOMAIN_G>,16020,1714829249771-splitting] in 3ms ... and after this follow the long lines from the procedure2.ProcedureExecutor 2024-05-06 22:17:11,837 INFO [PEWorker-16] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79}, {pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9}, {pid=18, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_51>, region=089a601094ea79ddbfd87a2572bb4f03}, {pid=19, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:zbl_persistent, region=ba37bf35a4430884d1a6ebbca95a8387}, {pid=20, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:grh_persistent, region=68531e269838537d534e81683e2a7097}, {pid=21, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_28>:<TABLE_28>, region=72f61b803e206cc95dc2a05b1112d39f}, {pid=22, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_37>:<TABLE_5>, region=f696c18f229b7b6525405d55afdd57cb}, {pid=23, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:grh_client_timeline, region=285ddc10b5d812318af616f7986ac6d1}, {pid=24, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_2>:<TABLE_51>, region=f56d6589a0660ea69432e3ff1bfa6941}, {pid=25, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_46>, region=0a0f5b07562c6220f4b15c8128748925}, {pid=26, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_30>:<TABLE_15>, region=d4568f96d953bd370840c8942c263d6d}, {pid=27, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_12>, region=7b8c2748761fe9da697b565bd51c29ff}, {pid=28, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_13>, region=911abd94189cd5cacad87054333c7756}, {pid=29, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_12>, region=d028ce2dae91e85c853dfbf24167b89f}, {pid=30, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_41>, region=907ba360d0a32358465ffa19b877d0ff}, {pid=31, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_8>:<TABLE_6>, region=07c3225b0f58be3adfe3e10f4054e5e8}, {pid=32, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_7>:<TABLE_31>, region=eba2fd53661e007937008e1b4b8723a2}, {pid=33, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_41>, region=96d4c009fb28c87f11371273343c86c2}, {pid=34, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>:<TABLE_53>, region=a1cc9bb5815382500ebc1f8ff2fe2889}, {pid=35, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_11>:<TABLE_45>, region=72af38e5137228c4a96c5afd1483141b}, {pid=36, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_31>:<TABLE_72>, region=2ae51a84d97bc465eca9b9d534585752}, {pid=37, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:tbb_persistent, region=60f28a92e81332855bc0bc8814b43c9c}, {pid=38, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.flowrun, region=1041abc1f0324a01072e30ac4651068f}, {pid=39, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:grh, region=913fc5fbecc382c9eb130bbca997a52a}, {pid=40, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:idp, region=52b26bbcb3470e2fda96a27cfb2bc728}, {pid=41, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:nbr, region=9d53deeb6951b5625c714f52f40cc2e2}, {pid=42, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=audit:ztw, region=3204e2b67fcf1ff91580c11f919c9261}, {pid=43, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_32>, region=f9459615516226d166054c69348c614c}, {pid=44, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_46>, region=cb9fe991b45b07aaca74b1b19cd1d8a9}, {pid=45, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_18>:<TABLE_24>, region=3c0ac9e61eb863ac6834645a45b2ce92}, {pid=46, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_15>: Employee, region=3152b0b94f3dab7872292d320681192f}, {pid=47, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_23>:<TABLE_29>, region=2c1fabd4bedd491f1d74f814ca788c24}, {pid=48, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=698bf8a9ad8970d1059273622b3beea9}, {pid=49, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=prod.timelineservice.application, region=7a54ab62ca00389083cacfd18859b19b}, {pid=50, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:ijs_persistent, region=9ce8fccc5665f0256fb8ab8e25618bcc}, {pid=51, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:ijs_persistent, region=fa3effb1e5c14db984279936ab111895}, {pid=52, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_63>, region=53d819a79ac851e839f5ff6a6b149b19}, {pid=53, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_16>:<TABLE_15>, region=5864381b7aedceae4ffae2d1eb84620e}, {pid=54, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_51>, region=39dc5b21757fb3745c703656d2385132}, {pid=55, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_63>, region=484aacabb3202a69d985c8316db6ed2e}, {pid=56, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=5aa11c337c3f916a35a27dea488e22d6}, {pid=57, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_20>:<TABLE_41>, region=971001559dbb1da0c4518773a5bd598f}, {pid=58, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_32>:<TABLE_72>, region=7d8e127283f4c0116ac780ad7274a419}, {pid=59, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_22>:<TABLE_33>, region=145a675885029dd31097cbd697243703}, {pid=60, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_24>:<TABLE_41>, region=4639b130e351fc822f26948d88ee50e1}, {pid=61, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_33>:<TABLE_53>, region=3e29eed2edc1b2abb1dbe7e416087835}, {pid=62, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_25>:<TABLE_7>, region=815ce30ee31cc91f4a6ae4b1ff787bad}] 2024-05-06 22:17:11,885 INFO [PEWorker-11] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=63, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:wlt_by_query, region=85d79840d75bd9d605e7f3fa9a330655}, {pid=64, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=stats:wlt_quantifier_index, region=9372d2b0f3629ed5c088dfc0aa541961}]*no* further _formatting_ is done here And first procedures reporting results? 2024-05-06 22:17:11,887 INFO [PEWorker-8] procedure.MasterProcedureScheduler: Took xlock for pid=17, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_6>:<TABLE_23>, region=9cceeb2adfe53ca9dff5eca04c7dbcd9 2024-05-06 22:17:11,889 INFO [PEWorker-7] procedure.MasterProcedureScheduler: Took xlock for pid=16, ppid=5, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=<NAMESPACE_10>:<TABLE_38>, region=2a6db1ba124e9cc69f8ae48ab6591a79

MrNicen · ‎05-07-2024

Somehow it was decided that for table hbase:meta the region should enter state RUNNABLE on RS l225-n2. Why l225-n2? Note, rit is still reported offline. 2024-05-06 22:16:38,408 INFO [PEWorker-14] procedure.MasterProcedureScheduler: Took xlock for pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 2024-05-06 22:16:38,417 INFO [PEWorker-14] assignment.AssignProcedure: Starting pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=l225-n2.<DOMAIN_G>,16020,1714828981130; forceNewPlan=false, retain=true target svr=null Here l227-n2 is calling in and Thread-16 reacts. 2024-05-06 22:16:38,536 INFO [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l227-n2.<DOMAIN_G>,16020,1715026585628 2024-05-06 22:16:38,555 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=1; waited=151ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms The following may be the first relevant WARN. Its is about ServerStateNode for l227-n2 being empty. Are these nodes persisted in Zookeeper? 2024-05-06 22:16:38,570 WARN [PEWorker-10] assignment.AssignmentManager: Why is ServerStateNode for l227-n2.<DOMAIN_G>,16020,1715026585628 empty at this point? Creating... 2024-05-06 22:16:38,570 INFO [PEWorker-10] assignment.AssignProcedure: Early suspend! pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=l227-n2.<DOMAIN_G>,16020,1715026585628 Next, all RSs are registering themselves and ephemeral node's are being created (I think in in ZK): 2024-05-06 22:16:38,579 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l227-n2.<DOMAIN_G>,16020,1715026585628] 2024-05-06 22:16:39,409 INFO [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l228-n1.<DOMAIN_G>,16020,1715026585682 2024-05-06 22:16:39,432 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l228-n1.<DOMAIN_G>,16020,1715026585682] 2024-05-06 22:16:39,457 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=2; waited=1053ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms 2024-05-06 22:16:40,100 INFO [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l224-n2.<DOMAIN_G>,16020,1715026584653 2024-05-06 22:16:40,108 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=3; waited=1704ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms 2024-05-06 22:16:40,112 INFO [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=16000] master.ServerManager: Registering regionserver=l230-n2.<DOMAIN_G>,16020,1715026584782 2024-05-06 22:16:40,115 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l226-n1.<DOMAIN_G>,16020,1715026584512 2024-05-06 22:16:40,116 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l226-n2.<DOMAIN_G>,16020,1715026584780 2024-05-06 22:16:40,120 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l230-n1.<DOMAIN_G>,16020,1715026584638 2024-05-06 22:16:40,132 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l224-n2.<DOMAIN_G>,16020,1715026584653] 2024-05-06 22:16:40,135 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l230-n2.<DOMAIN_G>,16020,1715026584782] 2024-05-06 22:16:40,135 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l226-n1.<DOMAIN_G>,16020,1715026584512] 2024-05-06 22:16:40,135 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l226-n2.<DOMAIN_G>,16020,1715026584780] 2024-05-06 22:16:40,142 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l230-n1.<DOMAIN_G>,16020,1715026584638] 2024-05-06 22:16:40,154 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l225-n1.<DOMAIN_G>,16020,1715026584877 2024-05-06 22:16:40,158 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=8; waited=1754ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms 2024-05-06 22:16:40,175 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l225-n1.<DOMAIN_G>,16020,1715026584877] 2024-05-06 22:16:40,323 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l224-n1.<DOMAIN_G>,16020,1715026584891 2024-05-06 22:16:40,343 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l224-n1.<DOMAIN_G>,16020,1715026584891] 2024-05-06 22:16:40,359 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=9; waited=1955ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms 2024-05-06 22:16:40,528 INFO [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000] master.ServerManager: Registering regionserver=l225-n2.<DOMAIN_G>,16020,1715026585040 2024-05-06 22:16:40,529 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l229-n1.<DOMAIN_G>,16020,1715026584835 2024-05-06 22:16:40,531 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l229-n2.<DOMAIN_G>,16020,1715026584960 2024-05-06 22:16:40,549 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l225-n2.<DOMAIN_G>,16020,1715026585040] 2024-05-06 22:16:40,551 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l229-n1.<DOMAIN_G>,16020,1715026584835] 2024-05-06 22:16:40,552 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l229-n2.<DOMAIN_G>,16020,1715026584960] 2024-05-06 22:16:40,559 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=12; waited=2155ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms 2024-05-06 22:16:40,840 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l227-n1.<DOMAIN_G>,16020,1715026585107 2024-05-06 22:16:40,861 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=13; waited=2457ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms 2024-05-06 22:16:40,864 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l227-n1.<DOMAIN_G>,16020,1715026585107] 2024-05-06 22:16:41,088 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.ServerManager: Registering regionserver=l228-n2.<DOMAIN_G>,16020,1715026585349 2024-05-06 22:16:41,112 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=2707ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms 2024-05-06 22:16:41,112 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [l228-n2.<DOMAIN_G>,16020,1715026585349] Hold on, we are almost reaching the point where the **bleep** hits the fan. The region of htable hbase:meta is dispatched to l227-n2, no mention anymore of rit=offline. But note that the RS has changed from l225-n2 to l227-n2, replicaid=0 in Zookeeper 2024-05-06 22:16:42,579 INFO [PEWorker-7] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2.<DOMAIN_G>,16020,1715026585628 2024-05-06 22:16:42,585 INFO [PEWorker-7] assignment.RegionTransitionProcedure: Dispatch pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 Server manager is awaiting even more RSs, but we have 14 and they are all registered at this point. 2024-05-06 22:16:42,615 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=4211ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-1504ms Now from PEWorker-8 Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2 and finally the ProcedureExecutor reports state=SUCCESS for AssignProcedure table=hbase:meta: 2024-05-06 22:16:43,572 INFO [PEWorker-8] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as l227-n2.<DOMAIN_G>,16020,1715026585628 2024-05-06 22:16:43,583 INFO [PEWorker-8] procedure2.ProcedureExecutor: Finished subprocedure pid=15, resume processing parent pid=8, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=l225-n2.<DOMAIN_G>,16020,1714828981130, splitWal=true, meta=true 2024-05-06 22:16:43,584 INFO [PEWorker-8] procedure2.ProcedureExecutor: Finished pid=15, ppid=8, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 5.1780sec, unfinishedSiblingCount=0 Time ticks on and no more region server registrations: 2024-05-06 22:16:44,119 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=5714ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-3007ms 2024-05-06 22:16:45,623 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=7219ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-4512ms No we arrive at the point where the problem manifests itself: 2024-05-06 22:16:45,644 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740 2024-05-06 22:16:45,644 WARN [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628 From her on I am skipping the INFO logs of the ServerManager Waiting on regionserver count=14. More WARNs: 2024-05-06 22:16:48,650 WARN [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED but no procedure found (complete?); set location=l227-n2.<DOMAIN_G>,16020,1715026585628 2024-05-06 22:16:51,655 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] assignment.AssignmentManager: META REPORTED: rit=OPEN, location=l227-n2.<DOMAIN_G>,16020,1715026585628, table=hbase:meta, region=1588230740 ... Several similar errors Than we arrive at master is running: 2024-05-06 22:17:08,175 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=14; waited=29771ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=-27064ms 2024-05-06 22:17:08,426 INFO [Thread-16] master.ServerManager: Finished waiting on RegionServer count=14; waited=30021ms, expected min=1 server(s), max=NO_LIMIT server(s), master is running Next we get loads these regions being assigend: 2024-05-06 22:17:08,614 INFO [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=bd9b9493492af8b42a99ff7e4cee8eb8, regionState=OPEN, lastHost=l230-n2.<DOMAIN_G>,16020,1714828981071, regionLocation=l230-n2.<DOMAIN_G>,16020,1714828981071, openSeqNum=531452 2024-05-06 22:17:08,615 INFO [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=de38ce1846144c1eb843ca9248efb791, regionState=OPEN, lastHost=l225-n2.<DOMAIN_G>,16020,1714828981130, regionLocation=l225-n2.<DOMAIN_G>,16020,1714828981130, openSeqNum=516441 2024-05-06 22:17:08,615 INFO [Thread-16] assignment.RegionStateStore: Load hbase:meta entry region=6546461582f9e25f18f5c77f6cf8fe6d, regionState=OPEN, lastHost=l225-n2.<DOMAIN_G>,16020,1714828981130, regionLocation=l225-n2.<DOMAIN_G>,16020,1714828981130, openSeqNum=403570 ...

MrNicen · ‎05-07-2024

Together with some colleagues, we have analysed the logs after restarting HBase, and made the following lay mans analysis of the last startup log: These seem innocent. HMaster starts en the RSs are not online yet: 2024-05-06 22:16:38,187 INFO [Thread-16] replication.ReplicationPeersZKImpl: Added peer cluster=l200-n4.<DOMAIN_M>,l201-n4.<DOMAIN_M>,l204-n4.<DOMAIN_M>:2181:/hbase-unsecure 2024-05-06 22:16:38,201 WARN [Thread-16] master.ServerManager: Server l224-n2.<DOMAIN_G>,16020,1659400236292 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n2.<DOMAIN_G>,16020,1659400236292] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l227-n2.<DOMAIN_G>,16020,1651336158836 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l227-n2.<DOMAIN_G>,16020,1651336158836] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n2.<DOMAIN_G>,16020,1713307724458 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n2.<DOMAIN_G>,16020,1713307724458] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l225-n1.<DOMAIN_G>,16020,1651396933254 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l225-n1.<DOMAIN_G>,16020,1651396933254] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l230-n2.<DOMAIN_G>,16020,1651396881056 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l230-n2.<DOMAIN_G>,16020,1651396881056] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n1.<DOMAIN_G>,16020,1660153454095 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n1.<DOMAIN_G>,16020,1660153454095] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l226-n2.<DOMAIN_G>,16020,1651397141435 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l226-n2.<DOMAIN_G>,16020,1651397141435] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l229-n1.<DOMAIN_G>,16020,1651336158866 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n1.<DOMAIN_G>,16020,1651336158866] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l228-n2.<DOMAIN_G>,16020,1651396801801 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l228-n2.<DOMAIN_G>,16020,1651396801801] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l227-n1.<DOMAIN_G>,16020,1651336158636 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l227-n1.<DOMAIN_G>,16020,1651336158636] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l224-n1.<DOMAIN_G>,16020,1651336158334 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l224-n1.<DOMAIN_G>,16020,1651336158334] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l226-n1.<DOMAIN_G>,16020,1651336158228 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l226-n1.<DOMAIN_G>,16020,1651336158228] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l229-n2.<DOMAIN_G>,16020,1651397094435 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,202 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n2.<DOMAIN_G>,16020,1651397094435] 2024-05-06 22:16:38,202 WARN [Thread-16] master.ServerManager: Server l228-n1.<DOMAIN_G>,16020,1651336158675 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l228-n1.<DOMAIN_G>,16020,1651336158675] 2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l230-n1.<DOMAIN_G>,16020,1651336158117 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l230-n1.<DOMAIN_G>,16020,1651336158117] 2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l225-n2.<DOMAIN_G>,16020,1651336158291 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l225-n2.<DOMAIN_G>,16020,1651336158291] 2024-05-06 22:16:38,203 WARN [Thread-16] master.ServerManager: Server l229-n1.<DOMAIN_G>,16020,1713308077605 is not currently online. Ignoring request to add it to draining list. 2024-05-06 22:16:38,203 INFO [Thread-16] master.DrainingServerTracker: Draining RS node created, adding to list [l229-n1.<DOMAIN_G>,16020,1713308077605] Here HMaster node l226-n3 becomes Active/primary and the SCPs are being started: 2024-05-06 22:16:38,203 INFO [Thread-16] master.HMaster: Active/primary master=l226-n3.<DOMAIN_G>,16000,1715026581275, sessionid=0x28f43c489256548, setting cluster-up flag (Was=true) 2024-05-06 22:16:38,282 INFO [Thread-16] master.HMaster: hbase:meta {1588230740 state=OPEN, ts=1715026596449, server=l225-n2.<DOMAIN_G>,16020,1714828981130} 2024-05-06 22:16:38,302 INFO [Thread-16] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1715026628302 2024-05-06 22:16:38,306 INFO [Thread-16] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=l228-n3.<DOMAIN_G>:2181,l229-n3.<DOMAIN_G>:2181,l230-n3.<DOMAIN_G>:2181 2024-05-06 22:16:38,306 INFO [Thread-16] zookeeper.ZooKeeper: Initiating client connection, connectString=l228-n3.<DOMAIN_G>:2181,l229-n3.<DOMAIN_G>:2181,l230-n3.<DOMAIN_G>:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@4e09d8ce 2024-05-06 22:16:38,316 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Opening socket connection to server l230-n3.<DOMAIN_G>/10.226.230.13:2181. Will not attempt to authenticate using SASL (unknown error) 2024-05-06 22:16:38,320 INFO [PEWorker-1] procedure.ServerCrashProcedure: Start pid=8, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l225-n2.<DOMAIN_G>,16020,1714828981130, splitWal=true, meta=true 2024-05-06 22:16:38,321 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.226.226.13:40378, server: l230-n3.<DOMAIN_G>/10.226.230.13:2181 2024-05-06 22:16:38,320 INFO [PEWorker-8] procedure.ServerCrashProcedure: Start pid=7, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l228-n1.<DOMAIN_G>,16020,1714828981415, splitWal=true, meta=false 2024-05-06 22:16:38,320 INFO [PEWorker-10] procedure.ServerCrashProcedure: Start pid=10, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l225-n1.<DOMAIN_G>,16020,1714829249747, splitWal=true, meta=false 2024-05-06 22:16:38,320 INFO [PEWorker-7] procedure.ServerCrashProcedure: Start pid=6, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l229-n2.<DOMAIN_G>,16020,1714828981440, splitWal=true, meta=false 2024-05-06 22:16:38,320 INFO [PEWorker-9] procedure.ServerCrashProcedure: Start pid=9, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l227-n2.<DOMAIN_G>,16020,1714829249989, splitWal=true, meta=false 2024-05-06 22:16:38,320 INFO [PEWorker-6] procedure.ServerCrashProcedure: Start pid=5, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l224-n1.<DOMAIN_G>,16020,1714828981244, splitWal=true, meta=false 2024-05-06 22:16:38,320 INFO [PEWorker-4] procedure.ServerCrashProcedure: Start pid=3, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l228-n2.<DOMAIN_G>,16020,1714828981422, splitWal=true, meta=false 2024-05-06 22:16:38,320 INFO [PEWorker-5] procedure.ServerCrashProcedure: Start pid=4, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l226-n2.<DOMAIN_G>,16020,1714828981235, splitWal=true, meta=false 2024-05-06 22:16:38,320 INFO [PEWorker-2] procedure.ServerCrashProcedure: Start pid=1, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l224-n2.<DOMAIN_G>,16020,1714828981192, splitWal=true, meta=false 2024-05-06 22:16:38,321 INFO [PEWorker-12] procedure.ServerCrashProcedure: Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l230-n2.<DOMAIN_G>,16020,1714828981071, splitWal=true, meta=false 2024-05-06 22:16:38,321 INFO [PEWorker-13] procedure.ServerCrashProcedure: Start pid=13, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l230-n1.<DOMAIN_G>,16020,1714828981173, splitWal=true, meta=false 2024-05-06 22:16:38,321 INFO [PEWorker-11] procedure.ServerCrashProcedure: Start pid=11, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l226-n1.<DOMAIN_G>,16020,1714829249771, splitWal=true, meta=false 2024-05-06 22:16:38,321 INFO [PEWorker-14] procedure.ServerCrashProcedure: Start pid=14, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l227-n1.<DOMAIN_G>,16020,1714828981420, splitWal=true, meta=false 2024-05-06 22:16:38,321 INFO [PEWorker-3] procedure.ServerCrashProcedure: Start pid=2, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=l229-n1.<DOMAIN_G>,16020,1714828981427, splitWal=true, meta=false Zookeeper connection: 2024-05-06 22:16:38,326 INFO [Thread-16-SendThread(l230-n3.<DOMAIN_G>:2181)] zookeeper.ClientCnxn: Session establishment complete on server l230-n3.<DOMAIN_G>/10.226.230.13:2181, sessionid = 0x38f43c48945656c, negotiated timeout = 60000 House keeping of WALs in Thread-16 2024-05-06 22:16:38,327 INFO [Thread-16] cleaner.CleanerChore: Cleaner pool size is 1 2024-05-06 22:16:38,328 INFO [Thread-16] cleaner.LogCleaner: Creating OldWALs cleaners with size=2 In PEWorker-9 more WAL management specifically related to l225-n2 (Why?): 2024-05-06 22:16:38,362 INFO [PEWorker-9] master.SplitLogManager: hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting is empty dir, no logs to split 2024-05-06 22:16:38,362 INFO [PEWorker-9] master.SplitLogManager: Started splitting 0 logs in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting] for [l225-n2.<DOMAIN_G>,16020,1714828981130] 2024-05-06 22:16:38,365 INFO [PEWorker-9] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://namenode-<CLUSTER_G>/apps/hbase/data/WALs/l225-n2.<DOMAIN_G>,16020,1714828981130-splitting] in 3ms In PEWorker-9 initialisation of the AssignProcedure 2024-05-06 22:16:38,397 INFO [PEWorker-9] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=15, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] Zookeeper Replication Peers, since we have replication enabled 2024-05-06 22:16:38,403 INFO [Thread-16] replication.ReplicationPeersZKImpl: Added peer cluster=l200-n4.<DOMAIN_M>,l201-n4.<DOMAIN_M>,l204-n4.<DOMAIN_M>:2181:/hbase-unsecure Still waiting for the first RS to call in: 2024-05-06 22:16:38,404 INFO [Thread-16] master.HMaster: Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1715026584479, completionTime=-1 2024-05-06 22:16:38,404 INFO [Thread-16] master.ServerManager: Waiting on regionserver count=0; waited=0ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=30000ms, lastChange=0ms

MrNicen · ‎05-07-2024

We tried the solution and the number of RIT went down to about 134. but still not zero. Most of the RS log's are "clean". In de hbase master log we get a nasty. 2024-05-06 22:34:22,745 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=l230-n2.SERVER,16020,1715026584782, table=<NAMESPACE>:<TABLE>, region=e1b0976c6c5d759............ and 2024-05-06 22:26:25,230 WARN [RpcServer.default.FPBQ.Fifo.handler=33,queue=3,port=16000] assignment.AssignmentManager: rit=OPENING, location=l225-n2.SERVER,16020,1715026585040, table=<NAMESPACE>:<TABLE>, region=d41e7923f187ee13d081591227c7805c reported OPEN on server=l225-n2.SERVER,16020,1715026585040 but state has otherwise AND NO procedure is running these lines are repeated ever x sec's and are sets off 134 lines i guess the same as the number of stuck regions I found out a can use hbck2 but have to add the "--skip version check option" *Note had some trouble with my accounts, so resetted the wrong one.

Online	Offline
Last Visited	‎01-15-2025 09:10 AM

Member Since	‎05-04-2024 11:42 PM
Last Visited	‎01-15-2025 09:10 AM
Posts	11
Kudos received	9

Cloudera Community

Tables not online

Re: Tables regios stuck in RIT

Re: Tables regios stuck in RIT

Re: Tables regios stuck in RIT

Re: Tables regios stuck in RIT

Re: Tables regios stuck in RIT

Re: Tables regios stuck in RIT

Re: Tables regios stuck in RIT

Re: Tables regios stuck in RIT

Re: Tables regios stuck in RIT