Hi, I’m currently experiencing an issue with inserting data into a Kudu table using Spark. Although I can successfully read the data via Hue, the insert operation through Spark is failing. Upon reviewing the tablet details for the table, I noticed an issue highlighted with the tablets, but I'm uncertain about the next steps to resolve it.
sudo -u kudu kudu cluster ksck tstewputil1
Connected to the Master Fetched info from all 3 Tablet Servers Tablet 0d811e918fd94b31aed4bfe337982ea8 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING 24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]
Tablet 1a27865b4f0c4dedaf43f490272c4f24 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING 24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]
Tablet 260af5f38836442f873d862d04de7615 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING 24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]
Tablet f738d1e1455044899230ead58c0a23e7 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING 24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]
Tablet 3f6347556e0649598f08edf4c5b9eaa1 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING 24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]
Tablet 255ed126c2fb46fcae4acc3eb3481a69 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING 24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]
Tablet b082a852dd40413f87cf3db126eb0f83 of table 'impala::mrs.NumberofSubs' is unavailable: 1 replica(s) not RUNNING 24483fcd36ce45d78d80beb04b3b0cf4: TS unavailable [LEADER]
Table impala::mrs.NumberofSubs has 7 unavailable tablet(s)
Could you please help me out from this insertion problem?
Replication factor seems to be 1 for above listed tables which is not a good setting given customer is facing failures and running into unavailability situation. This needs to be at least 3 in order to deal with transient errors or even a permanent data loss on a replica.
If the ksck report is showing every TS up and running, where is the problem now? Is there any tablet which is not accessible (for read/write) to the user?
You could list all the tablets using "kudu table list <master_addresses> --list_tablets" and search for the tablet server hosting the replica in question from the output; you can search for tablet uuid in old logs from the time when issue was seen, from there corresponding tablet server entry should have logged at some point.
But - generally speaking - if your KSCK is fixed and clean - let's say, it finished bootstrapping and/or copying after some time - and you still see INSERT problems from Spark - you need to open a case with Cloudera since the issue may require certain configs' evaluation and tuning.
It seem you're having an issue with unavailable tablets in your Kudu table, which is preventing Spark from inserting data. The output from kudu cluster ksck clearly shows the problem: the leader replicas for all tablets in the impala::mrs.NumberofSubs table are on a tablet server that's unavailable.
The key issue is that the tablet server with ID 24483fcd36ce45d78d80beb04b3b0cf4 is not running, and it happens to be the leader for all 7 tablets in your table.
Here's a solution to resolve this issue:
1. First, check the status of all Kudu tablet servers
5. After restarting the tablet server, wait a few minutes for it to rejoin the cluster and for leadership transitions to occur, then check the status again
If the tablet server is permanently lost or damaged, you'll need to recover the tablets a. Check if you have enough replicas (you should have at least 3 for production)
