Structured Unstructured Data for Pig and Hive

[#1]

Hi,

Can anyone elaborate on why pig and hive are better suited for unstructured and structured respectively?

My understanding of structured data is data that follows a particular schema and after that I've very little knowledge.

Is there a limitation with CSV files and variable length fields that Pig can handle easily?

[#2 · Accepted solution]

Pig is great for data discovery as you don't need schema on top of data, you can tell Pig how to consume the raw data by specifying delimeters. With Hive you still need to cleanse the data a bit before you can apply some sort of schema. So for dirty data, Pig is the first tool to use and then for familiar SQL functionality you can switch to Hive. Both can consume same datasets.

Hadoop is designed to be a generic data processing framework and for that it's designed so schema is applied at read stage as opposed to relational sources where schema is applied on write. Check out a few of the intro tutorials we have on Pig and Hive and you will see right away the concepts in action. its unlike anything you've worked with before.

[#3]

The limitation in Pig as far as CSV is concerned is that with default PigStorage loader function, it can handle a limited set of delimiters and corner cases. For wider array of cases, use CSVStorage loader function.

[#4]

good points; an example of some of the "corner cases" on CSV files (especially those generated by tools like Excel) are discussed in https://martin.atlassian.net/wiki/x/WYBmAQ.

[#5]

Can someone tell me a scenario where PIG is only option and a scenario where HIVE is only option?