Support Questions

Tomas79 · ‎01-03-2018

Hi,

I have a Kudu service running with a few tablets, but the INFO/WARNING logs on node went completely crazy, creating 10's of GB of log files.

The most of them contains warning information such as:

W0102 10:35:55.296466 58298 consensus_peers.cc:357] T d3c2d0eef63a4326af4fef61c9eaf0e9 
P f40241678ffd4ebaa8dcc39c58c1c0a7 -> Peer 605ae017362648d389f094d04296b8c7 
(ip-10-197-27-68.eu-west-1.compute.internal:7050): Couldn't send request to peer 
605ae017362648d389f094d04296b8c7 for tablet d3c2d0eef63a4326af4fef61c9eaf0e9.
 Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: 
FAILED: Not found: Can't find block: 0000000000779735. Retrying in the next heartbeat 
period. Already tried 2381364 times.

Now what I was suprised that Kudu is in green as a service and nobody complained about missing data.

But in logs there is a clear issue about missing block in a statement and the 2million's(!) attempt.

Few questions:

1. How to find out what is wrong in data, i.e. how to repair the missing block or find the corrupted table

2. How to set a treshold, because this retry with two million attempts seems to me very high

3. Is it possible to "detect" this issue in other way? Because the only symptom was a full disk in data node...

Thanks for the help,

Tomas

adar · ‎01-03-2018

1. This particular log message describes a missing block on a different server. If you look at the log for ip-10-197-27-68.eu-west-1.compute.internal:7050 you'll probably see a clearer message about the missing block. See if something happened to that server; maybe a disk went missing?
2. What CDH version are you using? Kudu is supposed to detect that this tablet is underreplicated because that replica isn't running, rereplicate the tablet elsewhere, then delete the broken replica. See https://issues.apache.org/jira/browse/KUDU-1407, which was fixed in Kudu 1.5.0, equivalent to CDH 5.13 or later.
3. The tserver web UI has a page which lists all tablets and breaks them down into those that are running and those that failed.

Tomas79 · ‎01-04-2018

The version of CDH is 5.11.1.

The logs in ip-10-197-27-68 contains few records about missing blocks, but it is hard to trace back which table has a problem, or which disks is missing (especially when everything in CM is green).

 5 E1219 15:45:46.582535 25050 tablet.cc:246] T afd9409992f7423e93816b500a69daa3 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(96): Not found: Can't find block: 0000000000016646
      6 E1219 15:45:46.582537 25049 tablet.cc:246] T 5b0bf82d322b4915bec1260a33cdb6b8 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(183): Not found: Can't find block: 0000000000187003
      7 E1219 15:45:46.582666 25048 ts_tablet_manager.cc:749] T d56103718d354fea8e10a95a164f2e75 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000781555
      8 E1219 15:45:46.582680 25050 ts_tablet_manager.cc:749] T afd9409992f7423e93816b500a69daa3 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000016646
      9 E1219 15:45:46.582726 25049 ts_tablet_manager.cc:749] T 5b0bf82d322b4915bec1260a33cdb6b8 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000187003
     10 E1219 15:45:46.584203 25048 tablet.cc:246] T 01c6c3abf85e4b1cb3767b173d60e401 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(1037): Not found: Can't find block: 0000000000046126
     11 E1219 15:45:46.584269 25048 ts_tablet_manager.cc:749] T 01c6c3abf85e4b1cb3767b173d60e401 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000046126
     12 E1219 15:45:46.585139 25048 tablet.cc:246] T b7562333c9e14322a03c4e562d0b0585 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(0): Not found: Can't find block: 0000000000057186
     13 E1219 15:45:46.585186 25048 ts_tablet_manager.cc:749] T b7562333c9e14322a03c4e562d0b0585 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000057186
     14 E1219 15:45:46.587437 25048 tablet.cc:246] T e989a401a3a44d1dae2c1143340a0676 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(3579): Not found: Can't find block: 0000000002669899
     15 E1219 15:45:46.587491 25048 ts_tablet_manager.cc:749] T e989a401a3a44d1dae2c1143340a0676 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000002669899
     16 E1219 15:45:46.588379 25048 tablet.cc:246] T 053bfd2aad4c44338f5beec6b81ace71 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(52358): Not found: Can't find block: 0000000004484875
     17 E1219 15:45:46.588420 25048 ts_tablet_manager.cc:749] T 053bfd2aad4c44338f5beec6b81ace71 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000004484875
     18 E1219 15:45:46.589145 25048 tablet.cc:246] T 4c6dbb80a8794624bc2eed2dcfa621e0 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(19): Not found: Can't find block: 0000000000779431
     19 E1219 15:45:46.589184 25048 ts_tablet_manager.cc:749] T 4c6dbb80a8794624bc2eed2dcfa621e0 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000779431
     20 E1219 15:45:46.589866 25048 tablet.cc:246] T 8b6455a77bb442b4bd26900c204d9820 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(2): Not found: Can't find block: 0000000000054998
     21 E1219 15:45:46.589905 25048 ts_tablet_manager.cc:749] T 8b6455a77bb442b4bd26900c204d9820 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000054998
     22 E1219 15:45:46.591459 25048 tablet.cc:246] T 6860b96404e74f4fa105a3c777a5aa5c P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(0): Not found: Can't find block: 0000000000054454
     23 E1219 15:45:46.591498 25048 ts_tablet_manager.cc:749] T 6860b96404e74f4fa105a3c777a5aa5c P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000054454
     24 E1219 15:45:46.592165 25048 tablet.cc:246] T c23b6df822fe47a4bb603e6599bc8fa4 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(2): Not found: Can't find block: 0000000000055006
     25 E1219 15:45:46.592203 25048 ts_tablet_manager.cc:749] T c23b6df822fe47a4bb603e6599bc8fa4 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000055006
     26 E1219 15:45:46.592854 25048 tablet.cc:246] T c334fe8add364c7aa3379e4890d07d12 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(51827): Not found: Can't find block: 0000000004559311
     27 E1219 15:45:46.592893 25048 ts_tablet_manager.cc:749] T c334fe8add364c7aa3379e4890d07d12 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000004559311
     28 E1219 15:45:46.644145 25049 tablet.cc:246] T 586f526030cb42dab79c60ecbf6e93fa P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(19): Not found: Can't find block: 0000000000779583
     29 E1219 15:45:46.644225 25049 ts_tablet_manager.cc:749] T 586f526030cb42dab79c60ecbf6e93fa P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000779583
     30 E1219 15:45:46.644315 25050 tablet.cc:246] T d3c2d0eef63a4326af4fef61c9eaf0e9 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(19): Not found: Can't find block: 0000000000779735
     31 E1219 15:45:46.644367 25050 ts_tablet_manager.cc:749] T d3c2d0eef63a4326af4fef61c9eaf0e9 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000779735
     32 E1219 15:45:46.645113 25050 tablet.cc:246] T c8884e56eaf146aa9110eb0cc57ef4bd P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(155): Not found: Can't find block: 0000000000018222
     33 E1219 15:45:46.645128 25049 tablet.cc:246] T 280b95e166bc4bfbaf3ff8f9f819e0b3 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(5341): Not found: Can't find block: 0000000002980443
     34 E1219 15:45:46.645154 25050 ts_tablet_manager.cc:749] T c8884e56eaf146aa9110eb0cc57ef4bd P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000018222
     35 E1219 15:45:46.645253 25049 ts_tablet_manager.cc:749] T 280b95e166bc4bfbaf3ff8f9f819e0b3 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000002980443
     36 E1219 15:45:46.647099 25048 tablet.cc:246] T 828ac02cb6c94a7085503a57f3a6bdfa P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(8): Not found: Can't find block: 0000000000782015
     37 E1219 15:45:46.647147 25048 ts_tablet_manager.cc:749] T 828ac02cb6c94a7085503a57f3a6bdfa P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000782015
     38 E1219 15:45:46.649103 25048 tablet.cc:246] T ed84f12cef7c48c089ed59add9076a32 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(0): Not found: Can't find block: 0000000000057178
     39 E1219 15:45:46.649144 25048 ts_tablet_manager.cc:749] T ed84f12cef7c48c089ed59add9076a32 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000057178
     40 E1219 15:45:46.649943 25048 tablet.cc:246] T 8ad88a00e203400b8bde11969d150821 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(116): Not found: Can't find block: 0000000000220075
     41 E1219 15:45:46.649982 25048 ts_tablet_manager.cc:749] T 8ad88a00e203400b8bde11969d150821 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000220075
     42 E1219 15:45:46.651765 25048 tablet.cc:246] T a04f95361a43439892c2a02bd587bd90 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(202): Not found: Can't find block: 0000000000188587
     43 E1219 15:45:46.651800 25048 ts_tablet_manager.cc:749] T a04f95361a43439892c2a02bd587bd90 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000188587
     44 E1219 15:45:46.652515 25048 tablet.cc:246] T ebe56433a0474716b69d4f198e2ee7bc P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(165): Not found: Can't find block: 0000000000187435
     45 E1219 15:45:46.652554 25048 ts_tablet_manager.cc:749] T ebe56433a0474716b69d4f198e2ee7bc P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000187435
     46 E1219 15:45:46.653415 25048 tablet.cc:246] T 2c9d1ab5e17e44e792e3e7dbecfcc11e P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(7): Not found: Can't find block: 0000000000781099
     47 E1219 15:45:46.653452 25048 ts_tablet_manager.cc:749] T 2c9d1ab5e17e44e792e3e7dbecfcc11e P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000781099
     48 E1219 15:45:46.654266 25048 tablet.cc:246] T 0e40ba72b1474d8790c6f3e3c26613ee P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(6): Not found: Can't find block: 0000000000780491
     49 E1219 15:45:46.654314 25048 ts_tablet_manager.cc:749] T 0e40ba72b1474d8790c6f3e3c26613ee P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000780491
     50 E1219 15:45:46.654999 25048 tablet.cc:246] T e4c8a716edfc466fbc8080bdcaae49ec P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(17): Not found: Can't find block: 0000000000779887
     51 E1219 15:45:46.655035 25048 ts_tablet_manager.cc:749] T e4c8a716edfc466fbc8080bdcaae49ec P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000779887
     52 E1219 15:45:46.655725 25048 tablet.cc:246] T 758af0f67dcf4ecdb9f8a41e59267de9 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(115): Not found: Can't find block: 0000000000219275
     53 E1219 15:45:46.655762 25048 ts_tablet_manager.cc:749] T 758af0f67dcf4ecdb9f8a41e59267de9 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000219275
     54 E1219 15:45:46.656723 25048 tablet.cc:246] T e0f609a373004af7a26528d5cb0ae576 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(0): Not found: Can't find block: 0000000000057182
     55 E1219 15:45:46.656759 25048 ts_tablet_manager.cc:749] T e0f609a373004af7a26528d5cb0ae576 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000057182
     56 E1219 15:45:46.658484 25048 tablet.cc:246] T dce34cd36973426288299ffad4ac84df P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(0): Not found: Can't find block: 0000000000055022
     57 E1219 15:45:46.658534 25048 ts_tablet_manager.cc:749] T dce34cd36973426288299ffad4ac84df P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000055022
     58 E1219 15:45:46.659234 25048 tablet.cc:246] T 044fec4e0a4f41adb2e5aed8dbaf7c8e P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(53079): Not found: Can't find block: 0000000004196159
     59 E1219 15:45:46.659272 25048 ts_tablet_manager.cc:749] T 044fec4e0a4f41adb2e5aed8dbaf7c8e P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000004196159
     60 E1219 15:45:46.660087 25048 tablet.cc:246] T ff3ecf83fe884b8d95cdf5d5382efee4 P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(18): Not found: Can't find block: 0000000000780003
     61 E1219 15:45:46.660125 25048 ts_tablet_manager.cc:749] T ff3ecf83fe884b8d95cdf5d5382efee4 P 605ae017362648d389f094d04296b8c7: Tablet failed to bootstrap: Not found: Can't find block: 0000000000780003
     62 E1219 15:45:46.702349 25050 tablet.cc:246] T 876a70c38eec4ec085487c1809e9b6bd P 605ae017362648d389f094d04296b8c7: Failed to open rowset RowSet(124): Not found: Can't find block: 0000000000217907

This is the list of all tables in Kudu:

Every table is in "Running" state.

Is it possible to do the fix via command line?

Upgrade to a newer version is in the plan, but till that time, I would like to fix it.

Tomas.

Support Questions

Kudu log files too big, too many exceptions attempts