Reply
Expert Contributor
Posts: 193
Registered: ‎07-01-2015

Kudu failed to load FS

Hi,

 after the upgrade from CDH 5.11 to 5.13 one Tablet server failed to start. This one is different from the others, because it has just three data directories (and disks) but the others have four disks. In the previous version (CDH 5.11) it wasnt a problem and the configuration was splitted to two configuration groups.

 

2:08:09.139 AM	INFO	cc:72	
Tablet server non-default flags:
--fs_data_dirs=/data/data4/kudu3/tablet2,/data/data5/kudu3/tablet2,/data/data6/kudu3/tablet2
--fs_wal_dir=/data/data4/kudu3/tabletwal2
--tserver_master_addrs=ip-10-197-18-248.eu-west-1.compute.internal,ip-10-197-23-140.eu-west-1.compute.internal,ip-10-197-23-43.eu-west-1.compute.internal
--heap_profile_path=/tmp/kudu-tserver.99324
--memory_limit_hard_bytes=17179869184
--flagfile=/run/cloudera-scm-agent/process/3701-kudu-KUDU_TSERVER/gflagfile
Tablet server version:
kudu 1.5.0-cdh5.13.1
revision 9044f1f377a6f2044d800600e5c2dc5e989c4dd4
build type RELEASE
built by jenkins at 09 Nov 2017 08:50:42 PST on impala-ec2-pkg-centos-7-0c27.vpc.cloudera.com
build id 2017-11-09_08-09-26
12:08:09.139 AM	INFO	cc:237	
Setting minidump size limit to 20M
12:08:09.141 AM	INFO	cc:79	
Initializing tablet server...
12:08:09.141 AM	INFO	cc:71	
NTP initialized. Skew: 500ppm Current error: 218983us
12:08:09.142 AM	INFO	cc:323	
Time spent opening directory manager: real 0.000s	user 0.000s	sys 0.000s
12:08:09.142 AM	FATAL	cc:80	
Check failed: _s.ok() Bad status: IO error: Failed to load FS layout: Could not verify integrity of files: /data/data4/kudu3/tablet2/data,/data/data5/kudu3/tablet2/data,/data/data6/kudu3/tablet2/data: 3 data directories provided, but expected 4

Now after the upgrade the Kudu instances are all in the same group. This is interesting because the configuration still holds one override for this specific host. I tried to manually reasssing this host to a specific group but still dont want to start.

 

image.pngAny help appreciated,

Thanks

 

Announcements