Reply
Highlighted
Contributor
Posts: 78
Registered: ‎06-19-2014

Previous batch had this exception

hi,

 

Insert data into kudu with mapreduce,I always got this exception,and the table has 3 replicas.

 

com.stumbleupon.async.TimeoutException: Timed out after 10000ms when joining Deferred@924595484(state=PENDING, result=null, callback=org.kududb.client.AsyncKuduSession$ConvertBatchToListOfResponsesCB@1ce2dd88 -> wakeup thread main, errback=passthrough -> wakeup thread main)
	at com.stumbleupon.async.Deferred.doJoin(Deferred.java:1183)
	at com.stumbleupon.async.Deferred.join(Deferred.java:1042)
	at org.kududb.client.KuduSession.flush(KuduSession.java:100)
	at shuyun.java.cds.mapreduce.ImportTradeMR$ImportReducer.reduce(ImportTradeMR.java:125)
	at shuyun.java.cds.mapreduce.ImportTradeMR$ImportReducer.reduce(ImportTradeMR.java:41)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
com.stumbleupon.async.TimeoutException: Timed out after 10000ms when joining Deferred@156122920(state=PENDING, result=null, callback=org.kududb.client.AsyncKuduSession$ConvertBatchToListOfResponsesCB@70145fdd -> wakeup thread main, errback=passthrough -> wakeup thread main)
	at com.stumbleupon.async.Deferred.doJoin(Deferred.java:1183)
	at com.stumbleupon.async.Deferred.join(Deferred.java:1042)
	at org.kududb.client.KuduSession.flush(KuduSession.java:100)
	at shuyun.java.cds.mapreduce.ImportTradeMR$ImportReducer.cleanup(ImportTradeMR.java:71)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-03-17 15:16:11,871 INFO [New I/O worker #24] org.kududb.client.AsyncKuduClient: Removing server 6a34f8ac4d974180a4b1c329702ecb0c from this tablet's cache 3c3af5355fa94095b50609db3f8e7849
2016-03-17 15:16:13,991 INFO [New I/O worker #26] org.kududb.client.AsyncKuduClient: Removing server 51564f83a1fe4fd99ef12bee55111e52 from this tablet's cache 4ae0273d8b3748fcb4d8c1c8eb11da49
2016-03-17 15:16:17,531 INFO [New I/O worker #27] org.kududb.client.AsyncKuduClient: Removing server b08a0d9a824843a1b50bcacf24165d8f from this tablet's cache 310428e7c2a34f3c954eb9410fccc873
2016-03-17 15:16:17,531 INFO [New I/O worker #27] org.kududb.client.AsyncKuduClient: Removing server b08a0d9a824843a1b50bcacf24165d8f from this tablet's cache dd54577eb3f6472da5ba8e4c102aee14
2016-03-17 15:16:18,431 INFO [New I/O worker #34] org.kududb.client.AsyncKuduClient: Removing server 6a34f8ac4d974180a4b1c329702ecb0c from this tablet's cache 3c3af5355fa94095b50609db3f8e7849
2016-03-17 15:16:18,431 ERROR [main] org.kududb.client.KuduSession: Previous batch had this exception
org.kududb.client.NonRecoverableException: Time out: KuduRpc(method=Write, tablet=3c3af5355fa94095b50609db3f8e7849, attempt=5, DeadlineTracker(timeout=10000, elapsed=12138), Deferred@1343287383(state=PENDING, result=null, callback=apply batch response -> callback: mark tablet 3c3af5355fa94095b50609db3f8e7849 inflight done -> (continuation of Deferred@135465170 after retry RPC@430446370) -> wakeup thread main -> FlushRetryCallback: retry flush tablet 3c3af5355fa94095b50609db3f8e7849 Batch(502 ops)@5a4a0b2a -> (continuation of Deferred@857940149 after retry RPC@1194878285), errback=apply batch error response -> errback: mark tablet 3c3af5355fa94095b50609db3f8e7849 inflight done -> (continuation of Deferred@135465170 after retry RPC@430446370) -> wakeup thread main -> FlushRetryErrback: retry flush tablet 3c3af5355fa94095b50609db3f8e7849 Batch(502 ops)@5a4a0b2a -> (continuation of Deferred@857940149 after retry RPC@1194878285)))
	at org.kududb.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:936)
	at org.kududb.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:1129)
	at org.kududb.client.AsyncKuduClient.handleRetryableError(AsyncKuduClient.java:1114)
	at org.kududb.client.AsyncKuduClient.handleTabletNotFound(AsyncKuduClient.java:1100)
	at org.kududb.client.TabletClient.failOrRetryRpc(TabletClient.java:664)
	at org.kududb.client.TabletClient.cleanup(TabletClient.java:626)
	at org.kududb.client.TabletClient.exceptionCaught(TabletClient.java:682)
	at org.kududb.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
	at org.kududb.client.TabletClient.handleUpstream(TabletClient.java:592)
	at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.kududb.client.shaded.org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:536)
	at org.kududb.client.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:236)
	at org.kududb.client.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler$ReadTimeoutTask$1.run(ReadTimeoutHandler.java:276)
	at org.kududb.client.shaded.org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40)
	at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
	at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
	at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.kududb.client.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.kududb.client.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.kududb.client.ConnectionResetException: [Peer 6a34f8ac4d974180a4b1c329702ecb0c] Connection reset on [id: 0xbc0094bf, /10.10.221.169:59755 => /10.10.212.183:7050]
	at org.kududb.client.TabletClient.cleanup(TabletClient.java:621)
	... 18 more

begging for help

 

regards,

 

rube

Cloudera Employee
Posts: 65
Registered: ‎09-28-2015

Re: Previous batch had this exception

Can you share more about your setup? How did you partition this table? How
many reducers do you have? Do you see anything in the tablet server log on
10.10.212.183 from around the same timestamp?

-Todd