Creating an HBase Coprocessor in Java

[#1]

I follow the examples from 3 Pillar Global Post and Apache Hbase Blog Post and then updated for newer versions.

To write an HBase Coprocessor you need Google Protocol Buffers. To use recent versions, on a Mac you need to install v2.50 as that works with HBase:

brew tap homebrew/versions
brew install protobuf250

Check your version protoc --version

Source Code with Maven Build (pom.xml) is here. You will need Maven and Java 7 (or newer) JDK for compilation.

Testing on Hadoop

export HADOOP_CLASSPATH=`hbase classpath`
hadoop jar hbasecoprocessor-1.0.jar com.dataflowdeveloper.hbasecoprocessor.SumEndPoint

Upload your Jar to HDFS

hadoop fs -mkdir /user/tspann 
hadoop fs -ls /user/tspann hadoop fs -put hbasecoprocessor-1.0.jar /user/tspann hadoop fs -chmod 777 /user/tspann/hbasecoprocessor-1.0.jar

Install Dynamically

disable 'stocks'
alter 'stocks',
'coprocessor'=>'hdfs://sandbox.hortonworks.com/user/tspann/hbasecoprocessor-1.0.jar|com.dataflowdeveloper.hbasecoprocessor.SumEndPoint|1001|arg1=1'
enable 'stocks'
describe 'stocks'

Testing Locally

java -classpath `hbase classpath`:hbasecoprocessor-1.0.jar com.dataflowdeveloper.hbasecoprocessor.SumEndPoint

Checking Table After Installation

[root@sandbox demo]# hbase shell
HBase Shell; enter
Version
1.1.2.2.4.0.0-169, r61dfb2b344f424a11f93b3f086eab815c1eb0b6a, Wed Feb 10
07:08:51 UTC 2016
hbase(main):001:0> describe 'stocks'
Table stocks is
ENABLED
stocks,
{TABLE_ATTRIBUTES => {coprocessor$1 =>
'hdfs://sandbox.hortonworks.com/user/tspann/hbasecoprocessor-1.0.jar|com.dataflowdeveloper.hbasecoprocessor.SumEndPoint|1001|arg1=1'} COLUMN FAMILIES
DESCRIPTION {NAME => 'cf',DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE
=> '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER',
MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
1 row(s) in 0.3270
seconds

You can see the coprocessor has been added and is enabled.

References

• https://github.com/larsgeorge/hbase-book/blob/master/ch04/src/main/java/coprocessor/RowCountEndpoint...
• https://github.com/Huawei-Hadoop/hindex
• https://github.com/apache/hbase/tree/branch-1.0/hbase-examples
• https://github.com/apache/hbase/blob/branch-1.0/hbase-examples/src/main/java/org/apache/hadoop/hbase...
• http://bigdatazone.blogspot.com/2015/05/hbase-coprocessor-using-protobuf-250.html
• http://hbase.apache.org/book.html#cp
• http://hbase.apache.org/book.html#cp_loading
• https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/package-summary.html
• http://hbase.apache.org/book.html#cp_example
• http://hbase.apache.org/xref/org/apache/hadoop/hbase/coprocessor/example/RowCountEndpoint.html
• https://github.com/apache/hbase/blob/branch-1.1/hbase-examples/pom.xml
• https://community.hortonworks.com/questions/2577/hbase-coprocessor-and-security.html
• https://www.3pillarglobal.com/insights/hbase-coprocessors#(endpoints-coprocessor)
• https://blogs.apache.org/hbase/entry/coprocessor_introduction
• https://github.com/dbist/HBaseUnitTest