Community Articles

Find and share helpful community-sourced technical articles.
Labels (1)
avatar

SYMPTOM: Hbase is giving an error KeyValue size too large when inserting large data sizes.

ERROR:

0java.lang.IllegalArgumentException: KeyValue size too large 
at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1521) 
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.validatePut(BufferedMutatorImpl.java:147) 
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:134) 
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:105) 
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1050) 
at org.apache.hadoop.hbase.rest.RowResource.update(RowResource.java:229) 
at org.apache.hadoop.hbase.rest.RowResource.put(RowResource.java:318) 
at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:497) 
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) 
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) 
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) 
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) 
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134) 
at 

ROOT CAUSE: hbase.client.keyvalue.maxsize is set too low

RESOLUTION: Set hbase.client.keyvalue.maxsize=0 This will allow the client key value to be allowed. Just be careful with this as too large of a keyvalue >1-2gb could have performance implications.

8,595 Views
0 Kudos