Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hbase closed upon write

Highlighted

hbase closed upon write

Contributor

I am on CDH 5.4.8 hbase 1.0

I am migrating to a new cluster and the data is sourced from CDH 5.3 hbase 0.98.6  

I took snapshots from the old cluster exported them to the new cluster - no errors

cloned the table, flush, major_compact and balanced the regions

the configs are practically identical to the old cluster

 

hbck and fsck all check out with no inconsistencies found

 

yet when we try to load data via thrift the regions start splitting at an outrageous pace.  

Eventually the regions become offline starting a snowball affect.  The region exits  are forcing the data to start transferring to other regions, with that creates large GC activity and multiple regions in transition before thrift can no longer execute any writes.  This usually happens after a few records.   How we put is that we perform rowkey scans before puts to ensure where we last left off. 

 

Thrift eventually stops writing due to regions being offline and no longer serving.  The regions eventually stop splitting and all the GC eventually complete.  Once things settle back down, I am able to perform hbck and fsck and it again comes back with no inconsistencies

 

I need ideas as to what is causing this

1 REPLY 1

Re: hbase closed upon write

Master Guru
Can you share the describe of the table? How much data was in the snapshot (per region)?

The only reason you should see a split is if the split policy is matched (usually a constant size policy, but can be different, describe will help tell).

The reason you may see splits occurring right upon further load is likely cause a major compact condition gets matched upon the flushes occurring as a result of the inserts.