11-10-2015 07:37 PM - edited 11-10-2015 08:10 PM
I am on CDH 5.4.8 hbase 1.0
I am migrating to a new cluster and the data is sourced from CDH 5.3 hbase 0.98.6
I took snapshots from the old cluster exported them to the new cluster - no errors
cloned the table, flush, major_compact and balanced the regions
the configs are practically identical to the old cluster
hbck and fsck all check out with no inconsistencies found
yet when we try to load data via thrift the regions start splitting at an outrageous pace.
Eventually the regions become offline starting a snowball affect. The region exits are forcing the data to start transferring to other regions, with that creates large GC activity and multiple regions in transition before thrift can no longer execute any writes. This usually happens after a few records. How we put is that we perform rowkey scans before puts to ensure where we last left off.
Thrift eventually stops writing due to regions being offline and no longer serving. The regions eventually stop splitting and all the GC eventually complete. Once things settle back down, I am able to perform hbck and fsck and it again comes back with no inconsistencies
I need ideas as to what is causing this
02-27-2016 09:04 PM