Geospatial Tutorial with GeoMesa Framework and HBase

[#1]

Introduction

GeoMesa is an Apache licensed open source suite of tools that enables large-scale geospatial analytics on cloud and distributed computing systems, letting you manage and analyze the huge spatio-temporal datasets that IoT, social media, tracking, and mobile phone applications seek to take advantage of today.

GeoMesa does this by providing spatio-temporal data persistence on top of the Accumulo, HBase, and Cassandra distributed column-oriented databases for massive storage of point, line, and polygon data. It allows rapid access to this data via queries that take full advantage of geographical properties to specify distance and area. GeoMesa also provides support for near real time stream processing of spatio-temporal data by layering spatial semantics on top of the Apache Kafka messaging system.

GeoMesa features include the ability to:

• Store gigabytes to petabytes of spatial data (tens of billions of points or more)
• Serve up tens of millions of points in seconds
• Ingest data faster than 10,000 records per second per node
• Scale horizontally easily (add more servers to add more capacity)
• Support Spark analytics
• Drive a map through GeoServer or other OGC Clients

Installation

GeoMesa supports traditional HBase installations as well as HBase running on Amazon’s EMR and Hortonworks’ Data Platform (HDP). For instructions on bootstrapping an EMR cluster, please read this tutorial: Bootstrapping GeoMesa HBase on AWS S3.

Tutorial Overview

The code in this tutorial only does a few things:

1. Establishes a new (static) SimpleFeatureType
2. Prepares the HBase table to store this type of data
3. Creates 1000 example SimpleFeatures
4. Writes these SimpleFeatures to the HBase table
5. Queries for a given geographic rectangle and time range, and attribute filter, and writes out the entries in the result set

Prerequisites

• Java Development Kit 1.8,
• Apache Maven,
• a GitHub client,
• HBase 1.2.x (optional), and
• GeoServer 2.9.1 (optional).

An existing HBase 1.1.x installation is helpful but not necessary. The tutorial described will work either with an existing HBase server or by downloading the HBase binary distribution and running it in "standalone" mode (described below).

GeoServer is only required for visualizing the HBase data. Setting up GeoServer is beyond the scope of this tutorial.

Download and Build the Tutorial

Clone the geomesa-tutorials distribution from GitHub:

$ git clone https://github.com/geomesa/geomesa-tutorials.git
$ cd geomesa-tutorials

The pom.xml file contains an explicit list of dependent libraries that will be bundled together into the final tutorial. You should confirm that the versions of HBase and Hadoop match what you are running; if it does not match, change the values of the hbase.version and hbase.hadoop.version properties. The version of GeoMesa that this tutorial targets matches the project version of the pom.xml. (Note that this tutorial has been tested with GeoMesa 1.2.2 or later).

The only reason these libraries are bundled into the final JAR is that this is easier for most people to do this than it is to set the classpath when running the tutorial. If you would rather not bundle these dependencies, mark them as provided in the POM, and update your classpath as appropriate.

GeoMesa's HBaseDataStore searches for a file called hbase-site.xml, which among other things configures the Zookeeper host(s) and port. If this file is not present on the classpath, the hbase-default.xml provided by hbase-common sets the default zookeeper quorum to "localhost" and port to 2181, which is what is used by the standalone HBase described in "Setting up HBase in standalone mode" above. If you have an existing HBase installation, you should copy your hbase-site.xml file into geomesa-quickstart-hbase/src/main/resources (or otherwise add it to the classpath when you run the tutorial).

To build the tutorial code:

$ cd geomesa-quickstart-hbase
$ mvn clean install

When this is complete, it should have built a JAR file that contains all of the code you need to run the tutorial.

Run the Tutorial

First, make sure that hbase.table.sanity.check property is set to false in hbase-site.xml

On the command line, run:

$ java -cp target/geomesa-quickstart-hbase-$VERSION.jar com.example.geomesa.hbase.HBaseQuickStart --bigtable_table_name geomesa

The only argument passed is the name of the HBase table where GeoMesa will store the feature type information. It will also create a table called <tablename>_<featuretype>_z3 which will store the Z3-indexed features. In our specific case, the table name will be geomesa_QuickStart_z3.

You should see output similar to the following (not including some of Maven's output and log4j's warnings), which lists the features that match the specified query in the tutorial do

Creating feature-type (schema):  QuickStart

Creating new features
Inserting new features
Submitting query
1.  Bierce|676|Fri Jul 18 08:22:03 EDT 2014|POINT (-78.08495724535888 37.590866849120395)|null
2.  Bierce|190|Sat Jul 26 19:06:19 EDT 2014|POINT (-78.1159944062711 37.64226959044015)|null
3.  Bierce|550|Mon Aug 04 08:27:52 EDT 2014|POINT (-78.01884511971093 37.68814732634964)|null
4.  Bierce|307|Tue Sep 09 11:23:22 EDT 2014|POINT (-78.18782181976381 37.6444865782879)|null
5.  Bierce|781|Wed Sep 10 01:14:16 EDT 2014|POINT (-78.0250604717695 37.58285696304815)|null

To see how the data is stored in HBase, use the HBase shell.

$ /path/to/hbase-1.2.3/bin/hbase shell

The type information is in the geomesa table (or whatever name you specified on the command line):

hbase> scan 'geomesa'
ROW  COLUMN+CELL  
QuickStart   column=M:schema, timestamp=1463593804724, value=Who:String,What:Long,When:Date,*Where:Point:s
  rid=4326,Why:String

The features are stored in <tablename>_<featuretype>_z3 (geomesa_QuickStart_z3 in this example):

hbase> scan 'geomesa_QuickStart_z3', { LIMIT => 3 }
ROW                              COLUMN+CELL                                                                                   \x08\xF7\x0F#\x83\x91\xAE\xA2\x column=D:\x0F#\x83\x91\xAE\xA2\xA8PObservation.452, timestamp=1463593805801, value=\x02\x00\x A8P                             00\x00@Observation.45\xB2Clemen\xF3\x01\x00\x00\x00\x00\x00\x00\x01\xC4\x01\x00\x00\x01CM8\x0                                 E\xA0\x01\x01\xC0S!\x93\xBCSg\x00\xC0CG\xBF$\x0DO\x7F\x80\x14\x1B$-?                          \x08\xF8\x06\x03\x19\xDFf\xA3p\ column=D:\x06\x03\x19\xDFf\xA3p\x0CObservation.362, timestamp=1463593805680, value=\x02\x00\x x0C                             00\x00@Observation.36\xB2Clemen\xF3\x01\x00\x00\x00\x00\x00\x00\x01j\x01\x00\x00\x01CQ\x17wh\                                 x01\x01\xC0S\x05\xA5b\xD49"\xC0B\x88*~\xD1\xA0}\x80\x14\x1B$-?                                \x08\xF8\x06\x07\x19S\xD0\xA21> column=D:\x06\x07\x19S\xD0\xA21>Observation.35, timestamp=1463593805664, value=\x02\x00\x00\x                                 00?Observation.3\xB5Clemen\xF3\x01\x00\x00\x00\x00\x00\x00\x00#\x01\x00\x00\x01CS?`x\x01\x01\                                
xC0S_\xA7+G\xADH\xC0B\x90\xEB\xF7`\xC2T\x80\x13\x1A#,>   

Recommendations

There are many reasons that GeoMesa can provide the best solution to your spatio-temporal database needs:

• You have Big Spatial Data sets and are reaching performance limitations of relational database systems. Perhaps you are looking at sharding strategies and wondering if now is the time to look for a new storage solution.
• You have very high-velocity data and need high read and write speeds.
• Your analytics operate in the cloud, perhaps using Spark, and you want to enable spatial analytics.
• You are looking for a supported, open-source alternative to expensive proprietary solutions.
• You are looking for a Platform as a Service (PaaS) database where you can store Big Spatial Data.
• You want to filter data using the rich Common Query Language (CQL) defined by the OGC.

Reference:

https://github.com/geomesa/geomesa-tutorials/tree/master/geomesa-quickstart-hbase

[#2]

I installed geomesa 1.3.5 in 10 node cluster. We are using kerberos to secure the cluster. Does geomesa works with kerberos?