HBase Cell level TTL does not work when after memstore flushed

[#1]

Hello ~

I'm looking for the solution of below symtoms.

Anyone who knows the solution ?

Thanks .

Environment : CDH 5.14 (HBase 1.2)

-- create test table
create 'jkkim','f'


-- put an sample data
put 'jkkim','row2','f:name','jonggyun Kim2'

-- put another sample data with TTL
put 'jkkim','row4','f:name','jonggyun Kim',{TTL => 10000}

-- scan after 15 seconds
scan 'jkkim'

-- row4 does not shown result (expected)
ROW COLUMN+CELL
row2 column=f:name, timestamp=1530609778641, value=jonggyun Kim2
1 row(s) in 0.0120 seconds


--put sample data again with TTL 
put 'jkkim','row4','f:name','jonggyun Kim',{TTL => 10000}

-- Run manually flush command
flush 'jkkim'


--scan after 15 seconds
scan 'jkkim'

-- row4 is shown result (not expected)
ROW COLUMN+CELL
row2 column=f:name, timestamp=1530609778641, value=jonggyun Kim2
row4 column=f:name, timestamp=1530609975940, value=jonggyun Kim

[#2 · Accepted solution]

Cell TTLs are a HFile V3 feature (as far as persistence goes).

CDH5 HBase uses HFile V2 by default for backward compatibility reasons with older CDH5 HBase versions. To persist features properly into HFiles, you must manually enable the HFile V3 feature.

You are likely missing the following configuration property:

<property>
  <name>hfile.format.version</name>
  <value>3</value>
</property>

This must be manually added into both the CM fields noted below:

• HBase - Configuration - 'HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml'
• HBase - Configuration - 'HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml'

Note that enabling this will not let you rollback your new HFiles of V3 back to V2 in future. That said, we do have a lot of users on V3 running without any issues.

[#3]

Dear Harsh J

Thank you very much. 

Very helpful.

One more questions about changing hfile format.

After change hfile format version, is it rewrite all of hfile to new version format?

Or Just be applied only newly created hfile?

Thanks.

[#4]

Immediately after the config change and restart, the existing HFiles will
stay as is (on V2), only newly flushed HFiles will be V3. But when the
table/region undergoes a major compaction, all HFiles will be rewritten to
V3. You can force a rewrite with the HBase Shell 'major_compact' command to
have it immediately rewrite all files.

[#5]

Got it!

Thank you for the explanation.