question Re: kudu Data length checksum does not match in Archives of Support Questions (Read Only)

kudu Data length checksum does not match

zppwxz — Fri, 16 Sep 2022 13:21:31 GMT

F0618 21:06:21.008426 28512 tablet_server_main.cc:80] Check failed: _s.ok() Bad status: Corruption: Failed to load FS layout: Could not process records in container /var/lib/kudu/tserver/data/332c0788fd874fa19c9f4fb0b862bfa7: Data length checksum does not match: Incorrect checksum in file /var/lib/kudu/tserver/data/332c0788fd874fa19c9f4fb0b862bfa7.metadata at offset 706423: Checksum does not match. Expected: 0. Actual: 1214729159

How to fix the problem after the server is out of power

Re: kudu Data length checksum does not match

Hao Hao — Mon, 25 Jun 2018 21:29:27 GMT

Looks like this is tracked and fixed in KUDU-2260.

Re: kudu Data length checksum does not match

zppwxz — Tue, 03 Jul 2018 05:50:16 GMT

CDH-5.15.0-1.cdh5.15.0.p0.21

+ exec /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/kudu/sbin/kudu-tserver --tserver_master_addrs=hdoop1 --flagfile=/run/cloudera-scm-agent/process/345-kudu-KUDU_TSERVER/gflagfile
F0703 13:46:42.503458 30033 tablet_server_main.cc:80] Check failed: _s.ok() Bad status: Corruption: Failed to load FS layout: Could not process records in container /var/lib/kudu/tserver/data/data/4655ed673cfc4f4e9d5789342eeec779: Data length checksum does not match: Incorrect checksum in file /var/lib/kudu/tserver/data/data/4655ed673cfc4f4e9d5789342eeec779.metadata at offset 6757669: Checksum does not match. Expected: 0. Actual: 1214729159
*** Check failure stack trace: ***
Wrote minidump to /var/log/kudu/minidumps/kudu-tserver/576b3747-215f-b988-5810abaf-1647ead9.dmp
*** Aborted at 1530596802 (unix time) try "date -d @1530596802" if you are using GNU date ***
PC: @     0x7fa653284c37 gsignal
*** SIGABRT (@0x3da00007551) received by PID 30033 (TID 0x7fa6555e5900) from PID 30033; stack trace: ***
    @     0x7fa6551d4330 (unknown)
    @     0x7fa653284c37 gsignal
    @     0x7fa653288028 abort
    @          0x1b6dee9 (unknown)
    @           0x91469d google::LogMessage::Fail()
    @           0x9166bc google::LogMessage::SendToLog()
    @           0x9141f9 google::LogMessage::Flush()
    @           0x91704f google::LogMessageFatal::~LogMessageFatal()
    @           0x8b455e (unknown)
    @     0x7fa65326ff45 __libc_start_main
    @           0x8b3e64 (unknown)

Re: kudu Data length checksum does not match

mpercy — Thu, 05 Jul 2018 17:44:26 GMT

You will have to use "dd" to remove the last record of the container file. The latest version of Kudu trunk (after 5.15) contains a --debug option to the "kudu pbc dump" tool that will tell you the offset of the file you should remove from the file, if you compile it.

If you can't compile Kudu from source to obtain that tool, then an easy option is to reformat the affected tablet server and start from scratch on that server, if you have additional replicas.

Another option is to use a hex editor to figure out the offset where there are is a run of 0s at the end of the file and truncate the 0s off of the file. Make sure to make a backup copy of the container metadata file first.

This will be prevented in a future release.

Re: kudu Data length checksum does not match

zppwxz — Mon, 09 Jul 2018 06:50:20 GMT

：） Thank you for your solution

Re: kudu Data length checksum does not match

mpercy — Mon, 09 Jul 2018 18:41:47 GMT

You're welcome. If that worked for you, please mark my response as the answer / solution to your question.