question Re: Hbase Storing pdf and Retrieval in Support Questions

Hbase Storing pdf and Retrieval

ashok_padmanabh — Tue, 19 Jul 2016 02:36:21 GMT

We are planning to store PDF and Word Documents in Hbase. Storing part is fine. Retrieval is part i have questions on.

1. If we need to query this - Is there a way to do it using any Reporting tools ? Hbase --> Hive External Table -->JDBC/ODBC --> Excel or any BI Tool. However how will the consumer app know that the field is a PDF FIle and not just a text field.

2. Is there a way for HBASE REST to handle this ?

Thanks in advance.

Re: Hbase Storing pdf and Retrieval

rgelhausen — Tue, 19 Jul 2016 02:56:45 GMT

Hi @Ash Pad, Phoenix has JDBC and REST APIs today. There is an ODBC driver under development which I believe is currently in beta. Thus you can do reporting style queries against whatever document metadata you store in "normal" column types.

To access the PDF object itself, you can use the JDBC/ODBC/REST apis to read/write the column as raw bytes. See the Phoenix DataTypes page to understand the various column types which support binary values.

Re: HBase REST- you could use this if desired, though I don't see why you would vs. using the built-in JDBC/ODBC capabilities.

Re: Hbase Storing pdf and Retrieval

tyu — Tue, 19 Jul 2016 04:01:45 GMT

Please take a look at:

https://issues.apache.org/jira/browse/HBASE-11339

which would reduce I/O amplification incurred by medium objects.

This feature is in the upcoming HDP 2.5 release.

Re: Hbase Storing pdf and Retrieval

sunile_manjee — Tue, 19 Jul 2016 09:26:57 GMT

@Ash Pad I personally like @Ted Yu answer. Until that is release I don't recommend storing these files on hbase. Instead a common practice to have the file save on HDFS and have the "pointer" stored in hbase.

Re: Hbase Storing pdf and Retrieval

rgelhausen — Tue, 19 Jul 2016 22:24:24 GMT

@Ash Pad, how big are your PDFs?

As in all things, it depends on your use case. If you PDFs are not in the multi-megabyte range, you may be fine storing them in a second column family today. This has the advantage of letting you query against doc metadata very quickly without needing to load full file contents into RegionServer memory. In most document management systems, this is highly desirable, as there is far more searching/querying than there is actual full content access.

Re: Hbase Storing pdf and Retrieval

ashok_padmanabh — Wed, 20 Jul 2016 00:51:24 GMT

PDFs are 50KB Max. and each rowkey can have upto a max of 5 PDFs associated with it. And the total volume of records would be around 500K range. Like you suggest we have 2 column Families, one for the metadata and one for the documents. your suggestion actually gives a vote of confidence to our thought process.

Re: Hbase Storing pdf and Retrieval

sabamu_lakhey — Fri, 16 Dec 2016 03:05:46 GMT

Can you please tell me how to store PDF and Word Documents in Hbase? @Ash Pad

Can you please tell me how did you store PDF and Word Documents in Hbase? @Ash Pad

Re: Hbase Storing pdf and Retrieval

tyu — Fri, 16 Dec 2016 03:13:57 GMT

HDP 2.5 has been released.

You can use the MOB feature now.