question Re: What's the best practice to get data from hbase and form dataframe for Python/R? in Archives of Support Questions (Read Only)

What's the best practice to get data from hbase and form dataframe for Python/R?

cui_lin — Wed, 16 Dec 2015 04:00:34 GMT

What's the best practice to get data from hbase and form dataframe for Python/R? If we want to use our Panda/R libraries, how to get data from hbase and form dataframe automatically?

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

pardeep_kumar — Thu, 17 Dec 2015 01:46:29 GMT

@Cui Lin I am not R guy but this would give you a good starting point depending on you want to use RevR, R or Python.

RHbase tutorials -->

https://github.com/RevolutionAnalytics/RHadoop/wik...

http://www.odbms.org/2015/06/intro-to-hbase-via-r-...

http://radar.oreilly.com/2014/08/scaling-up-data-f...

PandaHbase -->

https://github.com/livingstonese/pandas-hbase

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

cui_lin — Sat, 09 Jan 2016 02:21:46 GMT

It seems that the above can't satisfy all my need. What's the best way to get data out of hbase and save into files instead?

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

aervits — Sat, 09 Jan 2016 02:25:50 GMT

you can write Mapreduce program to dump data to files, you can use pig, you can use python with happybase, you have a lot of different options @Cui Lin.

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

cui_lin — Sat, 09 Jan 2016 02:30:37 GMT

I need to firstly run query to select records based on time, and then dump data into files or data frame. Happybase can't support query and its index has to be integer. Could you lead me some example on mapreduce or pig example?

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

aervits — Sat, 09 Jan 2016 02:34:51 GMT

@Cui Lin I updated my response above with links to mapreduce examples. You will need to setup a scanner based on your criteria and then run mapreduce to write out the data to files for Pig, here's an example to read data from Hbase table, then you just call "store data into 'location' using storage of your choice.

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

cui_lin — Sat, 09 Jan 2016 02:35:51 GMT

Is there any example to get data from Hbase using Spark in Hortonworks? MapR and Cloudera has some packages like this, not sure if it could work in Hortonworks.

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

aervits — Sat, 09 Jan 2016 02:38:59 GMT

there's work in progress to make Spark and HBase work efficiently together on the Hortonworks Side, we're not publishing anything until we can support it.

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

aervits — Sat, 09 Jan 2016 02:40:27 GMT

@Enis @vshukla @Devaraj Das @Ram Sriharsha

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

vshukla — Sat, 09 Jan 2016 03:04:28 GMT

We have an experimental Spark HBase connector, https://github.com/zhzhan/shc

With the following features

First class support for DataFrame API
JSON based catalog with rich data type support
Performant, scalable, enterprise-ready
Partition Pruning
Predicate Pushdown
Scan optimizations
Data Locality
Composite Rowkey
Leverage existing work in the HBase community

Please take look at the README of the above project.

Also see example https://github.com/zhzhan/shc/blob/master/src/main/scala/org/apache/spark/sql/execution/datasources/hbase/examples/HBaseSource.scala

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

sai_geetha — Thu, 28 Dec 2017 13:29:20 GMT

@Artem Ervits, Is there any progress on the Spark on HBase by Hortonworks. We are using the HDP platform but I am not able to easily conclude from the internet that confirms there is progress beyond the above discussion in 2016.

Re: What's the best practice to get data from hbase and form dataframe for Python/R?

aervits — Fri, 26 Jan 2018 00:38:26 GMT

@Sai Geetha M N please read our latest docs https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_spark-component-guide/content/ch_introduction-spark.html

and https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_spark-component-guide/content/ch08s05.html it's been out for a while now.