CompactionState.NONE : means no compaction is currently running. bq. And major compaction is an asychronous process, how to predict the wait time to get the state form the client API You need to keep on polling the API with some arbitrary wait time. ... View more

bq. 1) How to figure out the completion status for the compaction, since hbase client api majorCompact is an Asynchorous process you can use below API. CompactionState compactionState = admin.getCompactionState(table.getName()); 2) Is it mandatory to wait until compaction process completion , to query hbase for real time process Check your resources consumption with compaction as it impacts I/O, CPU usage and network . In standard server configuration , it is fine to run real time process during compaction. ... View more

It's better you declare every field as VARCHAR and then use functions to convert them to numbers[1] for mathematical operations. [1] https://phoenix.apache.org/language/functions.html#to_number ... View more

Do you have any errors in regionserver logs (it seems hbase:namespace table is not getting assigned some how). ... View more

Before running above command please take regionserver and master down if not already.(and keep the zookeeper running) ... View more

If it is not a production cluster(or not doing replication or other stuff dependent on zookeeper), can you try cleaning your zookeeper as it may be possible that there is znode for non-existent tables . bin/hbase clean --cleanZk ... View more

Map intermediate data will be written and sorted on local disk before sending to the reducer machines. You can reduce Map output Use Combiner in between Compressing it with Gzip to save network IO but there will be a tradeoff for CPU (mapred.compress.map.output=true, mapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec) Decrease split size(this will distribute map across server) and increase the number of reducers so that they have fewer amount of data to sort and process stop speculative execution( mapred.map.tasks.speculative.execution=false) If you can optimize on sorting, update algorithm for map.sort.class bq. Will i get any performance improvement if i increase io.sort.mb paramter when Map() task generates huge amount of data? Yes (but impact may not be huge), you can use with io.sort.factor ... View more

can you paste the complete stack trace. ... View more

you can use "major_compact" command to run a major compaction on the table. In HBase shell:- hbase(main):013:0> major_compact 'tablename' ... View more