Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is there a way to connect to HBase using C#?

avatar
Rising Star
 
1 ACCEPTED SOLUTION

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
10 REPLIES 10

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Mentor

avatar
Master Mentor

Here's another one https://hbasenet.codeplex.com/

avatar
Rising Star

hbasenet worked... thanks

avatar
Rising Star

The HDinsight SDK seems only to work with HD Insight on specific Hbase releases... 0.98,wonder if anyone had luck working it out on HDP or with hbase version 1+

avatar
Master Mentor

If you use REST it doesn't matter whether it's 0.98 or 1.x. Here is non HdInsight version http://hbase.apache.org/book.html#_rest

avatar
Rising Star

As has been mentioned in this thread, there is no native C# HBase client. however, there are several options for interacting with HBase from C#.

  1. C# HBase Thrift client - Thrift allows for defining service endpoints and data models in a common format and using code generators to create language specific bindings. HBase provides a Thirft server and definitions. There are many examples online for creating a C# HBase Thrift Client. hbase-thrift-csharp
  2. Marlin - Marlin is a C# client for interacting with Stargate (HBase REST API) that ultimately became hbase-sdk-for-net. I have not personally tested this against HBase 1.x+, but considering it uses Stargate, I expect it should work. If you are planning to use Stargate and implement your own client, which I would recommend over Thrift, make sure to use protobufs to avoid the JSON serialization overhead. Using a HTTP based approach also makes it much easier to load balance requests over multiple gateways.
  3. Phoenix Query Server - Phoenix is a SQL skin on HBase. Phoenix Query Server is a REST API for submitting SQL queries to Phoenix. Here is some example code, however, I have not yet tested it. hdinsight-phoenix-sharp
  4. Simba HBase ODBC Driver - Using ODBC to connect to HBase. I've heard positive feedback on this approach, especially from tools like Tableau. This is not open source and requires purchasing a license.
  5. Future: Phoenix ODBC Driver - I've been told a Phoenix ODBC driver is in the works. Unfortunately, no ETA.

What we really need is an Entity or LINQ based framework, as that's how C# developers expect to interact with backend data sources. At one point, a member of the community began developing Linq2Hive, but the project appears to be no more. It may be possible to leverage Linq2Hive and the HbaseStorageHandler, but that seems like a really poor pattern. 🙂

I'm sure there are others, but hopefully this helps.

avatar
Rising Star

Shane, thanks for the elaboration, I agree some further work should be done on this space... appreciate the elaboration again as this will be a great the explanation for whoever is trying to figure this out!

avatar
New Contributor

Regarding

https://github.com/hdinsight/hbase-sdk-for-net

https://github.com/Azure/hdinsight-phoenix-sharp

There are two modes if you read the source code,

gateway mode is specifically targeting HDInsight cluster or other cluster requires basic auth, which the requests will go through HDInsight cluster gateway and requires basic auth anthentication to forward the request.

VNet mode is by default used in Azure Virtual Network when deploying HDInsight cluster into it with your client VMs. But it also applies to all no-HDInsight and on-prem HBase clusters, as it sends requests directly to REST server endpoints, you can try the below sample code.

var scanOptions = RequestOptions.GetDefaultOptions();
scanOptions.Port = 8090;
scanOptions.AlternativeEndpoint = "/";
var restIPs = new List<string>();
restIPs.Add("10.0.0.15");
restIPs.Add("10.0.0.16");
var client = new HBaseClient(null, options, new LoadBalancerRoundRobin(restIPs));
var scanSettings = new Scanner { batch = 10 };
ScannerInformation scannerInfo = client.CreateScanner(testTableName, scanSettings, scanOptions);
var options = RequestOptions.GetDefaultOptions();
options.Port = 8090;
options.AlternativeEndpoint = "/";
options.AlternativeHost = scannerInfo.Location.Host;
client.DeleteScanner(testTableName, scannerInfo, options);

hbase-sdk-for-net applies to all HBase versions since it uses REST APIs.

hdinsight-phoenix-sharp only targets HDI 3.4 or later or other distributions as long as its Phoenix component supports Protobuf serialization.