question Re: Where to store a really wide table? in Archives of Support Questions (Read Only)

Where to store a really wide table?

TimothySpann — Sat, 11 Jun 2016 21:43:51 GMT

If you have 600+ columns and you need to access 20-30 columns at a time. What is the optimal type of storage:

Hive Table Stored as ORC with compression, vectorization and optimization; access with Tez and properly partition and bucket
HBase
HBase in Phoenix Table
Parquet File
AVRO File
Accumulo

Re: Where to store a really wide table?

tyu — Sat, 11 Jun 2016 22:11:08 GMT

You can choose hbase as storage.

HBase can easily handle hundreds of columns. Consider grouping the columns normally accessed together in the same column family.

Re: Where to store a really wide table?

tyu — Sat, 11 Jun 2016 22:11:39 GMT

If you can share your use case more, we would be able to provide more advice.

Re: Where to store a really wide table?

TimothySpann — Sat, 11 Jun 2016 23:02:41 GMT

600 columns of detailed information on customers with many kinds of attributes. The data needs to be access interactively in reports and through web applications. Access to a few hundred/thousand rows (plus summary information) from the dataset based on known 20-30 column chunks of related information bundles.

For interactive exploration of the data and extraction of this lists to use elsewhere.

Re: Where to store a really wide table?

tyu — Sun, 12 Jun 2016 08:42:17 GMT

HBase is a viable solution.

For query, consider Phoenix.

Re: Where to store a really wide table?

elserj — Mon, 13 Jun 2016 21:22:03 GMT

Accumulo would work for the same reasons that HBase does.