Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is there a way to find the completion of major compaction completion status via hbase client API?

avatar
Explorer

I want to trigger major compaction for all the tables in the hbase. Using hbase client API, major compaction is triggered for all the times via admin.majorCompact(tableName).
1) How to figure out the completion status for the compaction, since hbase client api majorCompact is an Asynchorous process.2) Is it mandatory to wait until compaction process completion , to query hbase for real time process

7 REPLIES 7

avatar

bq. 1) How to figure out the completion status for the compaction, since hbase client api majorCompact is an Asynchorous process

you can use below API.

CompactionState compactionState = admin.getCompactionState(table.getName());

2) Is it mandatory to wait until compaction process completion , to query hbase for real time process

Check your resources consumption with compaction as it impacts I/O, CPU usage and network . In standard server configuration , it is fine to run real time process during compaction.

avatar
Explorer

Hey Ankit, thanks for your response.
Hbase Client returns NONE though the compaction is completed for the table(which can be identifed from the logs).
And major compaction is an asychronous process, how to predict the wait time to get the state form the client API (In case if it returns the state properly)

avatar

CompactionState.NONE : means no compaction is currently running.

bq. And major compaction is an asychronous process, how to predict the wait time to get the state form the client API

You need to keep on polling the API with some arbitrary wait time.

avatar
Explorer

After polling for some time, if the state returns MAJOR, shall we confirm major compaction is completed?.

admin.majorCompact(TableName.valueOf("testTable"));

Thread.sleep("sometime");

CompactionState state=admin.getCompactionState("testTable");

if(state.equals(CompactionState.MAJOR))

{ //Major compaction is completed }

This code always returns state as NONE

avatar

NONE means no compaction is currently running (or completed) , if state return MAJOR , it means major compaction is still running. Either your table is small or you have a single file in each region resulting in major compaction to complete soon and showing state as NONE

avatar
Master Collaborator

bq. Thread.sleep("sometime");

How long was the sleep ?

I might be possible that compaction finished after the sleep.

Which HDP version are you using ?

avatar
Explorer

Hadoop: 2.7.3 , HBase 1.3.1
Major compaction has been triggered for all the tables in hbase via hbase client.

for(String table: listofTables)
admin.majorCompact(table)

Compaction has been triggered for all the tables which can be confirmed from hbase logs, but completion logs are not present for all the tables(checked logs even after two hours of compaction trigger)