Sometimes the WARNING "Negotiation complete: Network error: Server connection negotiation failed: server connection from 220.127.116.11:20391: BlockingRecv error: Recv() got EOF from remote (error 108)" occured when a lot of rows(about 200k rows/second) were inserted into kudu by storm. Then I restarted the storm job, the inserting job would be OK. No WARN/ERROR log appeared in storm.
I'm not familiar with Storm; how exactly does it write to Kudu? The error sounds like one piece of the Storm job abruptly disconnected from one of the Kudu servers. It may be harmless (maybe Storm will retry), or it may indicate a failure in the job. Given that Storm didn't exhibit any errors, it's likely to be the former, though you could always verify that by reading the data out of the table and ensuring that it's all there.
Is there any other associated logging in Kudu about this failure? Maybe a trace message of some kind?
I'm sorry, I mistake the storm log. There are many IOExeception in kudu client log, such as "[ERROR 2017-03-22 10:30:21 o.a.k.c.TabletClient:791 New I/O worker #1] [Peer 772d8abf66c24789a51ba852aea1b478] Unexpected exception from downstream on [id: 0x2b9826d2, /18.104.22.168:55922 => /22.214.171.124:7050] java.io.IOException: Broken pipe"
Are you pooling clients? Are you sure that your storm workers are not suffering GC pauses? It would be worth running the storm workers with verbose GC logging and check that you aren't seeing GC around the same time.
The server side log indicates that the workers are connecting and then taking 6-10 seconds to perform the necessary round trips for RPC connection negotiation. So, the client's disconnecting them.