Archives of Support Questions (Read Only)

sunile_manjee · ‎10-11-2016

what protocol is used for the new spark hbase connector? is spark using the hbase thrift?

mqureshi · ‎10-11-2016

Integration between Spark and HBase relies simply on HBaseContext which provides HBase configuration to Spark Executor. So, to answer which protocol is used, the answer is simple RPC. Please check following link for more details.

https://hbase.apache.org/book.html#spark

and here is the github link to HBase Spark module.

https://github.com/apache/hbase/tree/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark

View solution in original post

rgelhausen · ‎10-11-2016

@Sunile Manjee,

Depending on what data your Spark application attempts to access, it uses the relevant JVM HBase client APIs: filters, scans, gets, range-gets, etc.

See the code here.

mqureshi · ‎10-11-2016

@Sunile Manjee

Integration between Spark and HBase relies simply on HBaseContext which provides HBase configuration to Spark Executor. So, to answer which protocol is used, the answer is simple RPC. Please check following link for more details.

https://hbase.apache.org/book.html#spark

and here is the github link to HBase Spark module.

https://github.com/apache/hbase/tree/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark