question Re: WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation in Support Questions

WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation

rahul_gaikwad — Fri, 16 Sep 2022 14:15:29 GMT

While creating table , its failing with below error.

Query: CREATE TABLE REPORTKUDU
PRIMARY KEY(Marketing_Cloud_Visitor_ID)
PARTITION BY HASH(Marketing_Cloud_Visitor_ID) PARTITIONS 10
STORED AS KUDU TBLPROPERTIES('kudu.master_addresses'='X.X.X.X','kudu.num_tablet_replicas'='1')
AS SELECT * FROM REPORTN

WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation: required 52236317, size limit 10485760

Key already present in Kudu table 'impala::db.REPORTKUDU'. (1 of 9006 similar)

Re: WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation

Grant Henke — Fri, 29 Mar 2019 20:42:52 GMT

I suspect that large number of "Key already present" errors may play a part in the initial buffer size warning.

The `Key already present in Kudu table 'impala::db.REPORTKUDU'. (1 of 9006 similar)` is telling you that

you are trying to insert records which are not unique. The table you are selecting from must have multiple records for each `Marketing_Cloud_Visitor_ID`. I would suggest writing the create table statement to ensure the primary key is unique, or adjusting the select statment to deduplicate the records before inserting into Kudu.

Re: WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation

kunzhao — Tue, 18 Jan 2022 01:56:45 GMT

I encountered the same problem when performing insert operation. Have you solved this problem? thank you

Re: WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation

willx — Tue, 18 Jan 2022 14:30:56 GMT

Hi @rahul_gaikwad,

The issue occurs due to a known limitation. As the code points out, it indicates that the single write operation cannot fit into the configured maximum buffer size.Please refer to this KB:

https://my.cloudera.com/knowledge/quot-ERROR-Error-applying-Kudu-Op-Incomplete-buffer-size?id=302775

Regards,

Will

Re: WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation

kunzhao — Fri, 21 Jan 2022 01:52:43 GMT

Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)
-kudu_mutation_buffer_size=20971520
-kudu_error_buffer_size=20971520

Tablet Server Advanced Configuration Snippet (Safety Valve) for gflagfile
-max_cell_size_bytes=20971520

=========================

As above setted，It is working fine.

Thanks .