question Re: Using Spark Hbase in HDP 2.3.2 in Archives of Support Questions (Read Only)

Using Spark Hbase in HDP 2.3.2

pooja_khandelwa — Mon, 18 Jul 2016 21:20:53 GMT

I am using HDP 2.3.2 with spark 1.4.1. As per below link Spark hbase connector works with HDP 2.4.2 onwards. Can someone help with me how can i read hbase in spark using HDP 2.3.2

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_introduction-spark.html

Re: Using Spark Hbase in HDP 2.3.2

jyadav — Mon, 18 Jul 2016 21:30:10 GMT

Hi @pooja khandelwal As mentioned in the same doc link, spark 1.6.1 comes with spark hbase connector which under Technical preview(TP), moreover spark 1.6.1 is only certified on HDP 2.4.2. On the other side HDP 2.3.2 has spark 1.4.1 which doesn't support spark Hbase connector in HDP.

Re: Using Spark Hbase in HDP 2.3.2

pooja_khandelwa — Mon, 18 Jul 2016 21:30:11 GMT

I understand that. I am asking is there any other way I can read hbase inside spark?

Re: Using Spark Hbase in HDP 2.3.2

jyadav — Mon, 18 Jul 2016 21:35:51 GMT

The only recommended way is to upgrade the cluster to HDP2.4.2. You can also install spark 1.6.1 manually on HDP 2.3.2 but we don't recommend that.

Thanks.

Re: Using Spark Hbase in HDP 2.3.2

michel_weber — Thu, 21 Jul 2016 16:51:35 GMT

I'm reading from Hbase in a spark Hdp 2.3.4.0-3485 (and I've done reading with lower Hdp releases).

Read hbase through a RDD:

Configuration hbaseConfiguration = HBaseConfiguration.create();
hbaseConfiguration.set(TableInputFormat.INPUT_TABLE, "sometable");
JavaPairRDD<ImmutableBytesWritable, Result> hbaseRdd = sc.newAPIHadoopRDD(hbaseConfiguration, TableInputFormat.class, ImmutableBytesWritable.class, Result.class);

Or am I missing something?