Reply
New Contributor
Posts: 3
Registered: ‎11-09-2016

Kudu Client API upsert does not insert/update any data sometimes

I faced an issue when using the upsert operation to insert some rows  into the table,

sometimes,it  successfully  insert  some rows into table ,somes it  Insert failed ,but  the Kudu API upsert  does not throw any exception and does not insert/update any data.  The probability of failure is about 3 percent。

I set  AUTO_FLUSH_BACKGROUND mode  and  call KuduSession.getPendingErrors  to catch exception 。

 

The environment is:
CDH 6.11.0
Kudu 1.8
3 masters 8 tservers.

 

Cloudera Employee
Posts: 16
Registered: ‎04-25-2017

Re: Kudu Client API upsert does not insert/update any data sometimes

There's not enough information here to tell what's going on. Do you have an example that reproduces the problem or a minimal piece of code that mimics what you are doing in your application?
New Contributor
Posts: 3
Registered: ‎11-09-2016

Re: Kudu Client API upsert does not insert/update any data sometimes

create session code: KuduSession session = kuduClient.newSession(); session.setFlushMode(SessionConfiguration.FlushMode.AUTO_FLUSH_BACKGROUND); session.setFlushInterval(500); session.setMutationBufferSpace(500); catch Exception code: try { session.flush(); RowErrorsAndOverflowStatus error = session.getPendingErrors(); if (error.isOverflowed() || error.getRowErrors().length > 0) { if (error.isOverflowed()) { throw new CustomerException(ExceptionEnum.OPERATION_ERROR, "Kudu overflow exception occurred."); } StringBuilder errorMessage = new StringBuilder(); if (error.getRowErrors().length > 0) { for (RowError errorObj : error.getRowErrors()) { errorMessage.append(errorObj.toString()); errorMessage.append(";"); } } throw new CustomerException(ExceptionEnum.OPERATION_ERROR, "[" + errorMessage.toString() + "]"); } log.debug("Kudu operation success."); } catch (KuduException e) { log.error("Kudu exception while operation data.", e); throw new CustomerException(ExceptionEnum.OPERATION_ERROR, e); }
New Contributor
Posts: 3
Registered: ‎11-09-2016

Re: Kudu Client API upsert does not insert/update any data sometimes

[ Edited ]
My table description: create table uat_coofd.tb_depot_goods( depot_id bigint , waybill_number string , waybill_number_create_time bigint , depart_code string , Area_code_id bigint , node_type bigint , sub_waybill_number string , card_position_number string , incard_num bigint , import_export_sign bigint, transfered bigint, unloading_time bigint, loading_time bigint , incard_time bigint , create_time bigint , update_time bigint , network_type bigint, waybill_number_status bigint , export_line string , PRIMARY KEY (depot_id, waybill_number,waybill_number_create_time) ) PARTITION BY HASH (waybill_number) PARTITIONS 6, RANGE (waybill_number_create_time) ( PARTITION VALUES < 1556668800, PARTITION 1556668800 <= VALUES < 1559347200, PARTITION 1559347200 <= VALUES < 1561939200, PARTITION 1561939200 <= VALUES < 1564617600, PARTITION 1564617600 <= VALUES < 1567296000, PARTITION 1567296000 <= VALUES < 1569888000, PARTITION 1569888000 <= VALUES < 1572566400, PARTITION 1572566400 <= VALUES < 1575158400, PARTITION 1575158400 <= VALUES ) STORED AS KUDU ; What I'm doing: try { KuduSession session = kuduClient.newSession(); session.setFlushMode(SessionConfiguration.FlushMode.AUTO_FLUSH_BACKGROUND); session.setFlushInterval(500); session.setMutationBufferSpace(500); KuduTable table = kuduClient.openTable(tb_depot_goods); Upsert upst = table.newUpsert(); PartialRow row = upst.getRow(); row.addString("depot_id", 123); row.addString("waybill_number", "987654321"); row.addString("waybill_number_create_time",1554048000); ..... session.apply(upst); session.flush(); RowErrorsAndOverflowStatus error = session.getPendingErrors(); if (error.isOverflowed() || error.getRowErrors().length > 0) { if (error.isOverflowed()) { throw new CustomerException(ExceptionEnum.OPERATION_ERROR, "Kudu overflow exception occurred."); } StringBuilder errorMessage = new StringBuilder(); if (error.getRowErrors().length > 0) { for (RowError errorObj : error.getRowErrors()) { errorMessage.append(errorObj.toString()); errorMessage.append(";"); } } throw new CustomerException(ExceptionEnum.OPERATION_ERROR, "[" + errorMessage.toString() + "]"); } log.debug("Kudu operation success."); } catch (KuduException e) { log.error("Kudu exception while operation data.", e); throw new CustomerException(ExceptionEnum.OPERATION_ERROR, e); } I insert one million records in succession,Finally found that lost 300 records,the Kudu API does not throw any exception。