question Re: Read Hbase tables using Spark 2 in Support Questions

Read Hbase tables using Spark 2

ehanson — Wed, 03 Jan 2018 03:15:26 GMT

Looking for some suggestions on how to read HBase tables using Spark 2.2. I currently have HDP 2.6.3 installed and have been starting to use Spark 2.2. We have been using 1.6.3 with the spark hbase connector and that worked alright, but it doesn't seem to work with spark2. I also see a lot of references to using Phoenix, but that also doesn't support Spark2 until version 4.10 and HDP is still on 4.7. Does anyone have any suggestions or examples of how they are accomplishing interaction with HBase on Spark2?

Re: Read Hbase tables using Spark 2

dhyun — Wed, 03 Jan 2018 04:37:14 GMT

Hi, @Eric Hanson .
SHC seems to work for both Spark 1.6.3 and Spark 2.2. Could you share your specific problem with SHC here?

Re: Read Hbase tables using Spark 2

ehanson — Wed, 03 Jan 2018 04:50:58 GMT

@Dongjoon Hyun

What dependency are you using to get it to work with 2.2? I'm getting a missing or invalid dependency detected while loading class file HBaseContext.class. Looking at the hortonworks repo (http://repo.hortonworks.com/content/repositories/releases/) it looks like version 1.1.0.2.6.3.0-235 is built for Spark 2.2, but the matching hbase-spark dependency POM has Spark 2.1.1 as the spark version still. I'm guessing that's probably my issue, if you were able to get it to work, maybe I'm just doing something wrong.

Re: Read Hbase tables using Spark 2

dhyun — Wed, 03 Jan 2018 04:56:47 GMT

Let's ping the maintainer of SHC. Ping, @wyang Could you help @Eric Hanson?

Re: Read Hbase tables using Spark 2

ehanson — Wed, 10 Jan 2018 22:25:33 GMT

I created a hive table with HBase integration and was able to read from that table in my Spark job to resolve this for now

Re: Read Hbase tables using Spark 2

ehanson — Thu, 25 Jan 2018 04:30:18 GMT

@wyang Do you have any insight into why I can't get hbase-spark to work with Spark 2.2?

Re: Read Hbase tables using Spark 2

bikas — Fri, 26 Jan 2018 03:13:10 GMT

If you have HDP 2.6.3 then you should be able to find spark 2.2 version of spark-llap available under /usr/hdp/current/

Perhaps you are using the older versions of shc using --packages and thats not compatible with spark 2.2.

Re: Read Hbase tables using Spark 2

wyang — Fri, 26 Jan 2018 04:17:59 GMT

Hi Eric, Could you use this one http://repo.hortonworks.com/content/repositories/releases/com/hortonworks/shc/shc/1.1.0.2.6.3.13-5/ instead (which is for spark 2.2)?

Updated packages: http://repo.hortonworks.com/content/repositories/releases/com/hortonworks/shc/shc/

Thanks.