> Could it be related to region being split during the write ? If yes, why ? This is the right question to ask. The NotServingRegion exception occurs when the region for a given key is not online yet on a particular RS. Note that clients reach a specific RS by first querying meta, and if the sought RS claims its not serving the region then it could mean two things: (1) The region was recently moved around by the HBase Balancer, or (2) The region experienced a split and there are two parts in it now. In either case, the client treats the error as a cue to refetch meta content and then retry based on newer information present in it, which leads it to the newer location for the sought row key. Given your workload is insert and you only see this happen when you insert, its likely your problem is (2). Its easy to prove that by simply measuring how many regions were at the beginning (before the query runs) and how many there are at end (after the query completes or stalls) via the HMaster's Table UI or its equivalent page inside CM's HBase -> Tables area. Splitting is a good thing for scalability but it can certainly hamper your insert rate cause when the split is occurring all writes are temporarily rejected (this usually is only for a short period of time) and there is also an eventual major compaction that needs to occur later to truly split the singular region data store of the parent into two stores on HDFS (adds additional I/O expense). If by your observation of the numbers splits are indeed occurring, you can prevent them by reviewing and raising the split size attribute on the table, to have larger but fewer regions each. Splits are done usually when a size bound is hit. If your table is not pre-split, then consider pre-splitting it. The HBase documentation covers best practices around this: http://archive.cloudera.com/cdh5/cdh/5/hbase/book.html#manual_region_splitting_decisions (and other topics)
... View more
