question Re: Adding a master server in kudu in Support Questions

Adding a master server in kudu

RajeshBodolla — Fri, 16 Sep 2022 12:51:43 GMT

Hi,

I have installed kudu on my CDH 5.11.2 cluster with 2 masters. I want to add on emore master now. When i tried the regular way, my existing masters failed to start. Any thoughts are welcome.

Thanks

Re: Adding a master server in kudu

Hao Hao — Fri, 16 Feb 2018 21:55:37 GMT

Hi Rajesh,

Would you please be more specific about what are the error log you saw? Thanks!

Best,

Hao

Re: Adding a master server in kudu

RajeshBodolla — Mon, 19 Feb 2018 09:56:33 GMT

@Hao Hao

Error:

Unable to init master catalog manager: Invalid argument: Unable to initialize catalog manager: Failed to initialize sys tables async: on-disk and provided master lists are different.

Re: Adding a master server in kudu

awong — Tue, 20 Feb 2018 18:11:17 GMT

How did you add the master? Did you follow the steps outlined in the master
migration guide?

https://kudu.apache.org/docs/administration.html#migrate_to_multi_master

Re: Adding a master server in kudu

RajeshBodolla — Wed, 21 Feb 2018 09:16:54 GMT

Hi @awong

I added it like anyother role in cloudera manager.

Re: Adding a master server in kudu

RajeshBodolla — Wed, 21 Feb 2018 10:37:22 GMT

Hi @awong

I tried from the document you mentioned and have below exception while copying the data from existing masters.

kudu local_replica copy_from_remote --fs_wal_dir=/data/1/kudu/wal --fs_data_dirs=/data/1/kudu/data 00000000000000000000000000000000 xxxxxxxxxxx.migration.santanderuk.corp:7051
I0221 10:35:07.571077 37523 mem_tracker.cc:140] MemTracker: hard memory limit is 201.702087 GB
I0221 10:35:07.571144 37523 mem_tracker.cc:142] MemTracker: soft memory limit is 121.021248 GB
I0221 10:35:07.571871 37523 env_posix.cc:1308] Raising process file limit from 1024 to 4096
I0221 10:35:07.571907 37523 file_cache.cc:401] Constructed file cache lbm with capacity 1638
I0221 10:35:07.572870 37523 log_block_manager.cc:1343] Data dir /data/1/kudu/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0221 10:35:07.572883 37523 log_block_manager.cc:1353] Limiting containers on data directory /data/1/kudu/data/data to 2721 blocks
I0221 10:35:07.573066 37523 fs_manager.cc:256] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I0221 10:35:07.573223 37523 fs_manager.cc:259] Opened local filesystem: /data/1/kudu/data,/data/1/kudu/wal
uuid: "32cd48071fbe40ef8b4c8f2726d52962"
format_stamp: "Formatted at 2018-02-21 09:59:54 on xxxxxxxx"
I0221 10:35:07.577749 37523 tablet_copy_client.cc:166] T 00000000000000000000000000000000 P 32cd48071fbe40ef8b4c8f2726d52962: Tablet Copy client: Beginning tablet copy session from remote peer at address srvcloprlsk332.migration.santanderuk.corp:7051
Remote error: Unable to begin tablet copy session: Not authorized: unauthorized access to method: BeginTabletCopySession: Invalid argument: Unable to decode tablet copy RPC error message: message: "Not authorized: unauthorized access to method: BeginTabletCopySession" code: ERROR_APPLICATION

I'm running this as root user. My cluster is kerborized.

Re: Adding a master server in kudu

RajeshBodolla — Wed, 21 Feb 2018 10:58:22 GMT

I was able to sync the data after runnig the command as kudu user. But unfortunately, two tablet servers are not starting with below error.

Bad status: Already present: Could not create new FS layout: FSManager root is not empty: /data/7/kudu_ts/data

Re: Adding a master server in kudu

RajeshBodolla — Wed, 21 Feb 2018 11:25:31 GMT

Had to format the TS to get them started.