28804
DISCUSSIONS
102195
MEMBERS
3161
ARTICLES
Hi,
after the upgrade from CDH 5.11 to 5.13 one Tablet server failed to start. This one is different from the others, because it has just three data directories (and disks) but the others have four disks. In the previous version (CDH 5.11) it wasnt a problem and the configuration was splitted to two configuration groups.
2:08:09.139 AM INFO cc:72 Tablet server non-default flags: --fs_data_dirs=/data/data4/kudu3/tablet2,/data/data5/kudu3/tablet2,/data/data6/kudu3/tablet2 --fs_wal_dir=/data/data4/kudu3/tabletwal2 --tserver_master_addrs=ip-10-197-18-248.eu-west-1.compute.internal,ip-10-197-23-140.eu-west-1.compute.internal,ip-10-197-23-43.eu-west-1.compute.internal --heap_profile_path=/tmp/kudu-tserver.99324 --memory_limit_hard_bytes=17179869184 --flagfile=/run/cloudera-scm-agent/process/3701-kudu-KUDU_TSERVER/gflagfile Tablet server version: kudu 1.5.0-cdh5.13.1 revision 9044f1f377a6f2044d800600e5c2dc5e989c4dd4 build type RELEASE built by jenkins at 09 Nov 2017 08:50:42 PST on impala-ec2-pkg-centos-7-0c27.vpc.cloudera.com build id 2017-11-09_08-09-26 12:08:09.139 AM INFO cc:237 Setting minidump size limit to 20M 12:08:09.141 AM INFO cc:79 Initializing tablet server... 12:08:09.141 AM INFO cc:71 NTP initialized. Skew: 500ppm Current error: 218983us 12:08:09.142 AM INFO cc:323 Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s 12:08:09.142 AM FATAL cc:80 Check failed: _s.ok() Bad status: IO error: Failed to load FS layout: Could not verify integrity of files: /data/data4/kudu3/tablet2/data,/data/data5/kudu3/tablet2/data,/data/data6/kudu3/tablet2/data: 3 data directories provided, but expected 4
Now after the upgrade the Kudu instances are all in the same group. This is interesting because the configuration still holds one override for this specific host. I tried to manually reasssing this host to a specific group but still dont want to start.
Any help appreciated,
Thanks