Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Structured Unstructured Data for Pig and Hive

Solved Go to solution

Structured Unstructured Data for Pig and Hive

New Contributor

Hi,

Can anyone elaborate on why pig and hive are better suited for unstructured and structured respectively?

My understanding of structured data is data that follows a particular schema and after that I've very little knowledge.

Is there a limitation with CSV files and variable length fields that Pig can handle easily?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Structured Unstructured Data for Pig and Hive

Mentor

Pig is great for data discovery as you don't need schema on top of data, you can tell Pig how to consume the raw data by specifying delimeters. With Hive you still need to cleanse the data a bit before you can apply some sort of schema. So for dirty data, Pig is the first tool to use and then for familiar SQL functionality you can switch to Hive. Both can consume same datasets.

Hadoop is designed to be a generic data processing framework and for that it's designed so schema is applied at read stage as opposed to relational sources where schema is applied on write. Check out a few of the intro tutorials we have on Pig and Hive and you will see right away the concepts in action. its unlike anything you've worked with before.

4 REPLIES 4

Re: Structured Unstructured Data for Pig and Hive

Mentor

Pig is great for data discovery as you don't need schema on top of data, you can tell Pig how to consume the raw data by specifying delimeters. With Hive you still need to cleanse the data a bit before you can apply some sort of schema. So for dirty data, Pig is the first tool to use and then for familiar SQL functionality you can switch to Hive. Both can consume same datasets.

Hadoop is designed to be a generic data processing framework and for that it's designed so schema is applied at read stage as opposed to relational sources where schema is applied on write. Check out a few of the intro tutorials we have on Pig and Hive and you will see right away the concepts in action. its unlike anything you've worked with before.

Re: Structured Unstructured Data for Pig and Hive

Mentor

The limitation in Pig as far as CSV is concerned is that with default PigStorage loader function, it can handle a limited set of delimiters and corner cases. For wider array of cases, use CSVStorage loader function.

Re: Structured Unstructured Data for Pig and Hive

good points; an example of some of the "corner cases" on CSV files (especially those generated by tools like Excel) are discussed in https://martin.atlassian.net/wiki/x/WYBmAQ.

Re: Structured Unstructured Data for Pig and Hive

New Contributor

Can someone tell me a scenario where PIG is only option and a scenario where HIVE is only option?

Don't have an account?
Coming from Hortonworks? Activate your account here