question Re: hbase can't move data from disk after major_compact in Archives of Support Questions (Read Only)

hbase can't move data from disk after major_compact

136106445 — Mon, 12 Dec 2016 12:54:23 GMT

the flow variables value is defalt value. (hbase1.1.2 hdp2.3.4.0)

hbase.master.cleaner.interval

hbase.master.logcleaner.ttl

my hbase table is set 2 days ttl ,I do major_compact 'tablename' every day.

but 2 month later,I find hdfs /apps/hbase/data/data/default/tablename grows up every day.

why ? Please help me. Thanks very much

Re: hbase can't move data from disk after major_compact

mqureshi — Mon, 12 Dec 2016 13:49:20 GMT

@Xiaojie Ma

Can you check the setting for keep deleted cells fo your column family?

https://hbase.apache.org/book.html#cf.keep.deleted

Also, have you set the following?

hbase.hstore.time.to.purge.deletes

If yes, what's the value?

Re: hbase can't move data from disk after major_compact

136106445 — Mon, 12 Dec 2016 14:06:21 GMT

Both these two value which I didn't set,it may default values . But in my case, did these two values affect deletion ?

Re: hbase can't move data from disk after major_compact

136106445 — Mon, 12 Dec 2016 14:16:35 GMT

hbase.hstore.time.to.purge.deletes default is 0

Re: hbase can't move data from disk after major_compact

136106445 — Mon, 12 Dec 2016 14:17:50 GMT

I just want to move ttl timeout data away from my disk.

Re: hbase can't move data from disk after major_compact

136106445 — Mon, 12 Dec 2016 14:23:03 GMT

hbase.hstore.time.to.purge.deletes has been changed; if it is not set, or set to 0, all delete markers including those with future timestamp are purged during the later major compaction. Otherwise, a delete marker is kept until the major compaction after marker's timestamp + this setting.

I think these two value which have nothing to do in my case.

Re: hbase can't move data from disk after major_compact

mqureshi — Tue, 13 Dec 2016 00:07:02 GMT

Can you please share the result of following command: describe 'TABLE_NAME'

Re: hbase can't move data from disk after major_compact

mqureshi — Fri, 16 Dec 2016 01:28:47 GMT

@Xiaojie Ma

One thing just popped up in my mind. Do you have snapshots that may be pointing to your data? In that case, data is not deleted by major compaction and moved to an archiving folder. See under /hbase if you have .archive folder.

Re: hbase can't move data from disk after major_compact

136106445 — Tue, 20 Dec 2016 10:27:41 GMT

Thank you very much.

Re: hbase can't move data from disk after major_compact

136106445 — Tue, 27 Dec 2016 19:03:55 GMT

hbase(main):002:0> describe 'goods_new' Table goods_new is ENABLED goods_new COLUMN FAMILIES DESCRIPTION {NAME => 'v', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'TRUE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'LZO', MIN_VERSIONS => '0', BLOCKCACHE = > 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 1 row(s) in 0.2250 seconds hbase(main):003:0>

hbase api delete data very day.