question Re: How can you query using JSON against HDP? in Archives of Support Questions (Read Only)

How can you query using JSON against HDP?

mhendricks — Mon, 05 Oct 2015 23:17:22 GMT

The customer wants to use something like Apache Drill to query HDP using JSON due to the fact that it's self-describing.

Re: How can you query using JSON against HDP?

awatson — Mon, 05 Oct 2015 23:24:54 GMT

Take a look at Spark (and SparkSQL). It can automatically infer the schema of a JSON dataset

https://spark.apache.org/docs/1.4.1/sql-programming-guide.html#json-datasets

Re: How can you query using JSON against HDP?

deepesh1 — Mon, 05 Oct 2015 23:49:45 GMT

Apache Drill supports JSON as self describing data format, you can find the usage here. In Hive, HCatalog supports JSON as serde format for reading and writing data into tables.

Re: How can you query using JSON against HDP?

bsaini — Tue, 06 Oct 2015 02:02:08 GMT

One of the prospects recently evaluated Drill and while it worked for the structured / self-describing formats without creating schema, their experience was that the data type resolution aspect slowed the performance down. In any case, HWX does not support Drill officially so the on-us will be on customer to resolve any Drill related issues when using it with HDP.

On the other hand, my comment to customers is that Hive provides a consistent approach and in a way / semantics that is known to the database developers. Additionally, a larger community involvement and maturity of the product has hardened Hive over number of years.

JSONSerde is the easy to use way to handle JSON in HDP. In return of one time table creation, you get better performance as compared to Drill which does not seem like a bad trade off at all.