question Re: Structured Unstructured Data for Pig and Hive in Archives of Support Questions (Read Only)

Structured Unstructured Data for Pig and Hive

jgarrigan — Sat, 14 May 2016 17:32:34 GMT

Hi,

Can anyone elaborate on why pig and hive are better suited for unstructured and structured respectively?

My understanding of structured data is data that follows a particular schema and after that I've very little knowledge.

Is there a limitation with CSV files and variable length fields that Pig can handle easily?

Re: Structured Unstructured Data for Pig and Hive

aervits — Sat, 14 May 2016 18:45:11 GMT

Pig is great for data discovery as you don't need schema on top of data, you can tell Pig how to consume the raw data by specifying delimeters. With Hive you still need to cleanse the data a bit before you can apply some sort of schema. So for dirty data, Pig is the first tool to use and then for familiar SQL functionality you can switch to Hive. Both can consume same datasets.

Hadoop is designed to be a generic data processing framework and for that it's designed so schema is applied at read stage as opposed to relational sources where schema is applied on write. Check out a few of the intro tutorials we have on Pig and Hive and you will see right away the concepts in action. its unlike anything you've worked with before.

Re: Structured Unstructured Data for Pig and Hive

aervits — Sat, 14 May 2016 18:50:13 GMT

The limitation in Pig as far as CSV is concerned is that with default PigStorage loader function, it can handle a limited set of delimiters and corner cases. For wider array of cases, use CSVStorage loader function.

Re: Structured Unstructured Data for Pig and Hive

LesterMartin — Wed, 18 May 2016 03:51:22 GMT

good points; an example of some of the "corner cases" on CSV files (especially those generated by tools like Excel) are discussed in https://martin.atlassian.net/wiki/x/WYBmAQ.

Re: Structured Unstructured Data for Pig and Hive

arujit_das — Wed, 01 May 2019 22:18:14 GMT

Can someone tell me a scenario where PIG is only option and a scenario where HIVE is only option?