Reply
Highlighted
Contributor
Posts: 78
Registered: ‎06-19-2014

Tablet not found

hi,

kudu version is 0.7.0.

I use impala sql,insert into kudu table select * from textfile_table ,textfile_table has 30 columns and 2billions rows.

I got this exception:

Error while flushing Kudu session: 
Not found: Failed to write batch of 4 ops to tablet be53e89bef274a8699290db0d2a8ebda on tablet server 51564f83a1fe4fd99ef12bee55111e52 (spark-28:7050) after 2 attempt(s): Tablet not found
Not found: Failed to write batch of 4 ops to tablet be53e89bef274a8699290db0d2a8ebda on tablet server 51564f83a1fe4fd99ef12bee55111e52 (spark-28:7050) after 2 attempt(s): Tablet not found
Not found: Failed to write batch of 4 ops to tablet be53e89bef274a8699290db0d2a8ebda on tablet server 51564f83a1fe4fd99ef12bee55111e52 (spark-28:7050) after 2 attempt(s): Tablet not found
Not found: Failed to write batch of 4 ops to tablet be53e89bef274a8699290db0d2a8ebda on tablet server 51564f83a1fe4fd99ef12bee55111e52 (spark-28:7050) after 2 attempt(s): Tablet not found

Some configurations i missed?

 

Regards,

 

rube

 

 

Cloudera Employee
Posts: 65
Registered: ‎09-28-2015

Re: Tablet not found

Hi,

Can you please take a look at the tablet server log file on that node
(spark-28)? It should be in /var/log/kudu/kudu-tserver.INFO. See if you can
find anything about the tablet ID referenced in the error message.

Todd
Contributor
Posts: 78
Registered: ‎06-19-2014

Re: Tablet not found

tablet 514ba7efeb584fc18cc236ea852ee70e has the same exceptin:

 

the latest info log:

I0315 14:57:11.579435 23047 raft_consensus.cc:1483] T 514ba7efeb584fc18cc236ea852ee70e P c2d0a42d6f2944649b3bf54182c6c2a0 [term 42 FOLLOWER]: Raft consensus is shut down!
I0315 14:57:11.582353 23047 ts_tablet_manager.cc:947] T 514ba7efeb584fc18cc236ea852ee70e P c2d0a42d6f2944649b3bf54182c6c2a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I0315 14:57:11.789961 23042 ts_tablet_manager.cc:957] T 39722146864e499c9291bfde176be64d P c2d0a42d6f2944649b3bf54182c6c2a0: Tablet deleted. Last logged OpId: 3.14969
I0315 14:57:11.805245 23042 tablet_bootstrap.cc:363] T 39722146864e499c9291bfde176be64d P c2d0a42d6f2944649b3bf54182c6c2a0: Deleted tablet blocks from disk
I0315 14:57:11.890980 23071 raft_consensus.cc:799] T c01a5ad9bd91486badaf99860051d440 P c2d0a42d6f2944649b3bf54182c6c2a0 [term 325 FOLLOWER]: Rejecting Update request from peer b08a0d9a824843a1b50bcacf24165d8f
for earlier term 289. Current term is 325. Ops: []
I0315 14:57:11.901654 23060 raft_consensus.cc:777] T 5b707ffa362a4b06bc1642b1caf353fc P c2d0a42d6f2944649b3bf54182c6c2a0 [term 38 FOLLOWER]: Deduplicated request from leader. Original: 2.16613->[2.16614-2.17004
]   Dedup: 2.16639->[2.16640-2.17004]
I0315 14:57:11.901701 23060 raft_consensus.cc:799] T 5b707ffa362a4b06bc1642b1caf353fc P c2d0a42d6f2944649b3bf54182c6c2a0 [term 38 FOLLOWER]: Rejecting Update request from peer b08a0d9a824843a1b50bcacf24165d8f f
or earlier term 2. Current term is 38. Ops: [2.16614-2.16639]
I0315 14:57:11.907016 23047 ts_tablet_manager.cc:957] T 514ba7efeb584fc18cc236ea852ee70e P c2d0a42d6f2944649b3bf54182c6c2a0: Tablet deleted. Last logged OpId: 5.15401
I0315 14:57:11.921025 23047 tablet_bootstrap.cc:363] T 514ba7efeb584fc18cc236ea852ee70e P c2d0a42d6f2944649b3bf54182c6c2a0: Deleted tablet blocks from disk
I0315 14:57:11.932889 23065 raft_consensus.cc:799] T 48759fccb00c495999370a4635cfd5d3 P c2d0a42d6f2944649b3bf54182c6c2a0 [term 43 FOLLOWER]: Rejecting Update request from peer b08a0d9a824843a1b50bcacf24165d8f f
or earlier term 2. Current term is 43. Ops: []
Contributor
Posts: 78
Registered: ‎06-19-2014

Re: Tablet not found

It seams like,the leader tablet has been deleted, a follower is elected,and then ,can't find the older leader.
Contributor
Posts: 78
Registered: ‎06-19-2014

Re: Tablet not found

Can you help me?
Cloudera Employee
Posts: 65
Registered: ‎09-28-2015

Re: Tablet not found

Hi,

Sorry for the slow response. I've been traveling this week.

What version of Impala are you running? This looks like a bug that was
fixed in our more recent release. Can you paste the output of 'select
version();'? If it indicates that it was built in September 2015 it is the
older version that is missing this bug fix, and an update should fix the
issue.

-Todd
Contributor
Posts: 78
Registered: ‎06-19-2014

Re: Tablet not found

impalad version 2.3.0-IMPALA_KUDU-cdh5 RELEASE (build 95e755e6aea07cff03192fedc3f668dff5ae3bbe) Built on Mon, 29 Feb 2016 00:35:57 PST

CDH5.5.1, I think this is the latest version.

rube