Support Questions

Find answers, ask questions, and share your expertise

Using Spark Hbase in HDP 2.3.2

I am using HDP 2.3.2 with spark 1.4.1. As per below link Spark hbase connector works with HDP 2.4.2 onwards. Can someone help with me how can i read hbase in spark using HDP 2.3.2

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_introduction-spark...

1 ACCEPTED SOLUTION

Hi @pooja khandelwal As mentioned in the same doc link, spark 1.6.1 comes with spark hbase connector which under Technical preview(TP), moreover spark 1.6.1 is only certified on HDP 2.4.2. On the other side HDP 2.3.2 has spark 1.4.1 which doesn't support spark Hbase connector in HDP.

View solution in original post

4 REPLIES 4

Hi @pooja khandelwal As mentioned in the same doc link, spark 1.6.1 comes with spark hbase connector which under Technical preview(TP), moreover spark 1.6.1 is only certified on HDP 2.4.2. On the other side HDP 2.3.2 has spark 1.4.1 which doesn't support spark Hbase connector in HDP.

I understand that. I am asking is there any other way I can read hbase inside spark?

The only recommended way is to upgrade the cluster to HDP2.4.2. You can also install spark 1.6.1 manually on HDP 2.3.2 but we don't recommend that.

Thanks.

New Contributor

I'm reading from Hbase in a spark Hdp 2.3.4.0-3485 (and I've done reading with lower Hdp releases).

Read hbase through a RDD:

Configuration hbaseConfiguration = HBaseConfiguration.create();
hbaseConfiguration.set(TableInputFormat.INPUT_TABLE, "sometable");
JavaPairRDD<ImmutableBytesWritable, Result> hbaseRdd = sc.newAPIHadoopRDD(hbaseConfiguration, TableInputFormat.class, ImmutableBytesWritable.class, Result.class);


Or am I missing something?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.