Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Using Spark Hbase in HDP 2.3.2

Solved Go to solution
Highlighted

Using Spark Hbase in HDP 2.3.2

I am using HDP 2.3.2 with spark 1.4.1. As per below link Spark hbase connector works with HDP 2.4.2 onwards. Can someone help with me how can i read hbase in spark using HDP 2.3.2

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_introduction-spark...

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Using Spark Hbase in HDP 2.3.2

Hi @pooja khandelwal As mentioned in the same doc link, spark 1.6.1 comes with spark hbase connector which under Technical preview(TP), moreover spark 1.6.1 is only certified on HDP 2.4.2. On the other side HDP 2.3.2 has spark 1.4.1 which doesn't support spark Hbase connector in HDP.

View solution in original post

4 REPLIES 4
Highlighted

Re: Using Spark Hbase in HDP 2.3.2

Hi @pooja khandelwal As mentioned in the same doc link, spark 1.6.1 comes with spark hbase connector which under Technical preview(TP), moreover spark 1.6.1 is only certified on HDP 2.4.2. On the other side HDP 2.3.2 has spark 1.4.1 which doesn't support spark Hbase connector in HDP.

View solution in original post

Highlighted

Re: Using Spark Hbase in HDP 2.3.2

I understand that. I am asking is there any other way I can read hbase inside spark?

Highlighted

Re: Using Spark Hbase in HDP 2.3.2

The only recommended way is to upgrade the cluster to HDP2.4.2. You can also install spark 1.6.1 manually on HDP 2.3.2 but we don't recommend that.

Thanks.

Highlighted

Re: Using Spark Hbase in HDP 2.3.2

New Contributor

I'm reading from Hbase in a spark Hdp 2.3.4.0-3485 (and I've done reading with lower Hdp releases).

Read hbase through a RDD:

Configuration hbaseConfiguration = HBaseConfiguration.create();
hbaseConfiguration.set(TableInputFormat.INPUT_TABLE, "sometable");
JavaPairRDD<ImmutableBytesWritable, Result> hbaseRdd = sc.newAPIHadoopRDD(hbaseConfiguration, TableInputFormat.class, ImmutableBytesWritable.class, Result.class);


Or am I missing something?

Don't have an account?
Coming from Hortonworks? Activate your account here