Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kudu Client API upsert does not insert/update any data sometimes

Kudu Client API upsert does not insert/update any data sometimes

New Contributor

I faced an issue when using the upsert operation to insert some rows  into the table,

sometimes,it  successfully  insert  some rows into table ,somes it  Insert failed ,but  the Kudu API upsert  does not throw any exception and does not insert/update any data.  The probability of failure is about 3 percent。

I set  AUTO_FLUSH_BACKGROUND mode  and  call KuduSession.getPendingErrors  to catch exception 。

 

The environment is:
CDH 6.11.0
Kudu 1.8
3 masters 8 tservers.

 

3 REPLIES 3

Re: Kudu Client API upsert does not insert/update any data sometimes

Cloudera Employee
There's not enough information here to tell what's going on. Do you have an example that reproduces the problem or a minimal piece of code that mimics what you are doing in your application?
Highlighted

Re: Kudu Client API upsert does not insert/update any data sometimes

New Contributor
create session code: KuduSession session = kuduClient.newSession(); session.setFlushMode(SessionConfiguration.FlushMode.AUTO_FLUSH_BACKGROUND); session.setFlushInterval(500); session.setMutationBufferSpace(500); catch Exception code: try { session.flush(); RowErrorsAndOverflowStatus error = session.getPendingErrors(); if (error.isOverflowed() || error.getRowErrors().length > 0) { if (error.isOverflowed()) { throw new CustomerException(ExceptionEnum.OPERATION_ERROR, "Kudu overflow exception occurred."); } StringBuilder errorMessage = new StringBuilder(); if (error.getRowErrors().length > 0) { for (RowError errorObj : error.getRowErrors()) { errorMessage.append(errorObj.toString()); errorMessage.append(";"); } } throw new CustomerException(ExceptionEnum.OPERATION_ERROR, "[" + errorMessage.toString() + "]"); } log.debug("Kudu operation success."); } catch (KuduException e) { log.error("Kudu exception while operation data.", e); throw new CustomerException(ExceptionEnum.OPERATION_ERROR, e); }

Re: Kudu Client API upsert does not insert/update any data sometimes

New Contributor
My table description: create table uat_coofd.tb_depot_goods( depot_id bigint , waybill_number string , waybill_number_create_time bigint , depart_code string , Area_code_id bigint , node_type bigint , sub_waybill_number string , card_position_number string , incard_num bigint , import_export_sign bigint, transfered bigint, unloading_time bigint, loading_time bigint , incard_time bigint , create_time bigint , update_time bigint , network_type bigint, waybill_number_status bigint , export_line string , PRIMARY KEY (depot_id, waybill_number,waybill_number_create_time) ) PARTITION BY HASH (waybill_number) PARTITIONS 6, RANGE (waybill_number_create_time) ( PARTITION VALUES < 1556668800, PARTITION 1556668800 <= VALUES < 1559347200, PARTITION 1559347200 <= VALUES < 1561939200, PARTITION 1561939200 <= VALUES < 1564617600, PARTITION 1564617600 <= VALUES < 1567296000, PARTITION 1567296000 <= VALUES < 1569888000, PARTITION 1569888000 <= VALUES < 1572566400, PARTITION 1572566400 <= VALUES < 1575158400, PARTITION 1575158400 <= VALUES ) STORED AS KUDU ; What I'm doing: try { KuduSession session = kuduClient.newSession(); session.setFlushMode(SessionConfiguration.FlushMode.AUTO_FLUSH_BACKGROUND); session.setFlushInterval(500); session.setMutationBufferSpace(500); KuduTable table = kuduClient.openTable(tb_depot_goods); Upsert upst = table.newUpsert(); PartialRow row = upst.getRow(); row.addString("depot_id", 123); row.addString("waybill_number", "987654321"); row.addString("waybill_number_create_time",1554048000); ..... session.apply(upst); session.flush(); RowErrorsAndOverflowStatus error = session.getPendingErrors(); if (error.isOverflowed() || error.getRowErrors().length > 0) { if (error.isOverflowed()) { throw new CustomerException(ExceptionEnum.OPERATION_ERROR, "Kudu overflow exception occurred."); } StringBuilder errorMessage = new StringBuilder(); if (error.getRowErrors().length > 0) { for (RowError errorObj : error.getRowErrors()) { errorMessage.append(errorObj.toString()); errorMessage.append(";"); } } throw new CustomerException(ExceptionEnum.OPERATION_ERROR, "[" + errorMessage.toString() + "]"); } log.debug("Kudu operation success."); } catch (KuduException e) { log.error("Kudu exception while operation data.", e); throw new CustomerException(ExceptionEnum.OPERATION_ERROR, e); } I insert one million records in succession,Finally found that lost 300 records,the Kudu API does not throw any exception。