Kudo table not upserting data

[#1]

Hi, I’m currently experiencing an issue with inserting data into a Kudu table using Spark. Although I can successfully read the data via Hue, the insert operation through Spark is failing. Upon reviewing the tablet details for the table, I noticed an issue highlighted with the tablets, but I'm uncertain about the next steps to resolve it.

sudo -u kudu kudu cluster ksck tstewputil1

Connected to the Master
Fetched info from all 3 Tablet Servers
Tablet 0d811e918fd94b31aed4bfe337982ea8 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING
24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]

Tablet 1a27865b4f0c4dedaf43f490272c4f24 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING
24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]

Tablet 260af5f38836442f873d862d04de7615 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING
24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]

Tablet f738d1e1455044899230ead58c0a23e7 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING
24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]

Tablet 3f6347556e0649598f08edf4c5b9eaa1 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING
24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]

Tablet 255ed126c2fb46fcae4acc3eb3481a69 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING
24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]

Tablet b082a852dd40413f87cf3db126eb0f83 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING
24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]

Table impala::mrs.NumberofSubs has 7 unavailable tablet(s)

Could you please help me out from this insertion problem?

Thank you for your support

[#2]

Hi-

There are number of things to note here:

1. Replication factor seems to be 1 for above listed tables which is not a good setting given customer is facing failures and running into unavailability situation. This needs to be at least 3 in order to deal with transient errors or even a permanent data loss on a replica.
2. If the ksck report is showing every TS up and running, where is the problem now? Is there any tablet which is not accessible (for read/write) to the user?
3. You could list all the tablets using "kudu table list <master_addresses> --list_tablets" and search for the tablet server hosting the replica in question from the output; you can search for tablet uuid in old logs from the time when issue was seen, from there corresponding tablet server entry should have logged at some point.
   
   But - generally speaking - if your KSCK is fixed and clean - let's say, it finished bootstrapping and/or copying after some time - and you still see INSERT problems from Spark - you need to open a case with Cloudera since the issue may require certain configs' evaluation and tuning.

[#3]

@zeeshanmcs 

It seem you're having an issue with unavailable tablets in your Kudu table, which is preventing Spark from inserting data. The output from kudu cluster ksck clearly shows the problem: the leader replicas for all tablets in the impala::mrs.NumberofSubs table are on a tablet server that's unavailable.

The key issue is that the tablet server with ID 24483fcd36ce45d78d80beb04b3b0cf4 is not running, and it happens to be the leader for all 7 tablets in your table.

Here's a solution to resolve this issue:

1. First, check the status of all Kudu tablet servers

Spoiler

sudo systemctl status kudu-tserver

2. Look specifically for the tablet server with ID 24483fcd36ce45d78d80beb04b3b0cf4

Spoiler

sudo -u kudu kudu tserver list tstewputil1

3. If the tablet server is down, start it.

Spoiler

sudo systemctl start kudu-tserver

4. If the tablet server is running but not responding, restart it

Spoiler

sudo systemctl restart kudu-tserver

5. After restarting the tablet server, wait a few minutes for it to rejoin the cluster and for leadership transitions to occur, then check the status again

Spoiler

sudo -u kudu kudu cluster ksck tstewputil1

If the tablet server is permanently lost or damaged, you'll need to recover the tablets
a. Check if you have enough replicas (you should have at least 3 for production)

Spoiler

sudo -u kudu kudu table describe impala::mrs.NumberofSubs tstewputil1

b. If you have other healthy replicas, you can delete the failed server from the cluster and Kudu will automatically recover

Spoiler

sudo -u kudu kudu tserver delete tstewputil1 <tablet_server_uuid>

c. If this is the only replica and you don't have backups, you may need to:

• Create a new table with the same schema
• Load data from your source systems
• Or restore from a backup if available

If, after restarting, you still have issues, the problem might be:

• Disk space issues on the tablet server
• Configuration problems
• Network connectivity problems between servers

Check the Kudu tablet server logs for more details

Spoiler

less /var/log/kudu/kudu-tserver.log

Once the tablet server is back online and healthy, your Spark job should be able to insert data into the table successfully

Happy hadooping