- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Is there a way to find the completion of major compaction completion status via hbase client API?
- Labels:
-
Apache HBase
Created ‎09-13-2017 06:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I want to trigger major compaction for all the tables in the hbase. Using hbase client API, major compaction is triggered for all the times via admin.majorCompact(tableName).
1) How to figure out the completion status for the compaction, since hbase client api majorCompact is an Asynchorous process.2) Is it mandatory to wait until compaction process completion , to query hbase for real time process
Created ‎09-13-2017 06:15 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
bq. 1) How to figure out the completion status for the compaction, since hbase client api majorCompact is an Asynchorous process
you can use below API.
CompactionState compactionState = admin.getCompactionState(table.getName());
2) Is it mandatory to wait until compaction process completion , to query hbase for real time process
Check your resources consumption with compaction as it impacts I/O, CPU usage and network . In standard server configuration , it is fine to run real time process during compaction.
Created ‎09-13-2017 07:12 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey Ankit, thanks for your response.
Hbase Client returns NONE though the compaction is completed for the table(which can be identifed from the logs).
And major compaction is an asychronous process, how to predict the wait time to get the state form the client API (In case if it returns the state properly)
Created ‎09-13-2017 07:18 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
CompactionState.NONE : means no compaction is currently running.
bq. And major compaction is an asychronous process, how to predict the wait time to get the state form the client API
You need to keep on polling the API with some arbitrary wait time.
Created ‎09-13-2017 08:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
After polling for some time, if the state returns MAJOR, shall we confirm major compaction is completed?.
admin.majorCompact(TableName.valueOf("testTable"));
Thread.sleep("sometime");
CompactionState state=admin.getCompactionState("testTable");
if(state.equals(CompactionState.MAJOR))
{ //Major compaction is completed }
This code always returns state as NONE
Created ‎09-13-2017 08:29 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
NONE means no compaction is currently running (or completed) , if state return MAJOR , it means major compaction is still running. Either your table is small or you have a single file in each region resulting in major compaction to complete soon and showing state as NONE
Created ‎09-13-2017 06:23 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
bq. Thread.sleep("sometime");
How long was the sleep ?
I might be possible that compaction finished after the sleep.
Which HDP version are you using ?
Created ‎09-14-2017 01:20 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hadoop: 2.7.3 , HBase 1.3.1
Major compaction has been triggered for all the tables in hbase via hbase client.
for(String table: listofTables)
admin.majorCompact(table)
Compaction has been triggered for all the tables which can be confirmed from hbase logs, but completion logs are not present for all the tables(checked logs even after two hours of compaction trigger)
