Adding a master server in kudu

[#1]

Hi,

I have installed kudu on my CDH 5.11.2 cluster with 2 masters. I want to add on emore master now. When i tried the regular way, my existing masters failed to start. Any thoughts are welcome.

Thanks

[#2]

Hi Rajesh,

Would you please be more specific about what are the error log you saw? Thanks!

Best,

Hao

[#3]

@Hao Hao

Error:

Unable to init master catalog manager: Invalid argument: Unable to initialize catalog manager: Failed to initialize sys tables async: on-disk and provided master lists are different.

[#4]

How did you add the master? Did you follow the steps outlined in the master
migration guide?

https://kudu.apache.org/docs/administration.html#migrate_to_multi_master

[#5]

Hi @awong

I added it like anyother role in cloudera manager.

[#6]

Hi @awong

I tried from the document you mentioned and have below exception while copying the data from existing masters.

kudu local_replica copy_from_remote --fs_wal_dir=/data/1/kudu/wal --fs_data_dirs=/data/1/kudu/data 00000000000000000000000000000000 xxxxxxxxxxx.migration.santanderuk.corp:7051
I0221 10:35:07.571077 37523 mem_tracker.cc:140] MemTracker: hard memory limit is 201.702087 GB
I0221 10:35:07.571144 37523 mem_tracker.cc:142] MemTracker: soft memory limit is 121.021248 GB
I0221 10:35:07.571871 37523 env_posix.cc:1308] Raising process file limit from 1024 to 4096
I0221 10:35:07.571907 37523 file_cache.cc:401] Constructed file cache lbm with capacity 1638
I0221 10:35:07.572870 37523 log_block_manager.cc:1343] Data dir /data/1/kudu/data/data is on an ext4 filesystem vulnerable to KUDU-1508 with block size 4096
I0221 10:35:07.572883 37523 log_block_manager.cc:1353] Limiting containers on data directory /data/1/kudu/data/data to 2721 blocks
I0221 10:35:07.573066 37523 fs_manager.cc:256] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I0221 10:35:07.573223 37523 fs_manager.cc:259] Opened local filesystem: /data/1/kudu/data,/data/1/kudu/wal
uuid: "32cd48071fbe40ef8b4c8f2726d52962"
format_stamp: "Formatted at 2018-02-21 09:59:54 on xxxxxxxx"
I0221 10:35:07.577749 37523 tablet_copy_client.cc:166] T 00000000000000000000000000000000 P 32cd48071fbe40ef8b4c8f2726d52962: Tablet Copy client: Beginning tablet copy session from remote peer at address srvcloprlsk332.migration.santanderuk.corp:7051
Remote error: Unable to begin tablet copy session: Not authorized: unauthorized access to method: BeginTabletCopySession: Invalid argument: Unable to decode tablet copy RPC error message: message: "Not authorized: unauthorized access to method: BeginTabletCopySession" code: ERROR_APPLICATION

I'm running this as root user. My cluster is kerborized.

[#7]

I was able to sync the data after runnig the command as kudu user. But unfortunately, two tablet servers are not starting with below error.

Bad status: Already present: Could not create new FS layout: FSManager root is not empty: /data/7/kudu_ts/data

[#8 · Accepted solution]

Had to format the TS to get them started.