question Re: Faster and Better Optimized Storage format in Pig? in Archives of Support Questions (Read Only)

Faster and Better Optimized Storage format in Pig?

adnanalvee — Wed, 01 Mar 2017 00:30:34 GMT

Is there any better storage format for pig? Lets say I want to store a very large filtered hive table/data before any further processing. Is there any format that makes processing faster?

Re: Faster and Better Optimized Storage format in Pig?

aervits — Wed, 01 Mar 2017 00:36:46 GMT

@Adnan Alvee

use ORC format with HCatalog integration in Pig, take a look at my article https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.html

Re: Faster and Better Optimized Storage format in Pig?

adnanalvee — Fri, 10 Mar 2017 00:28:32 GMT

Wow. ORC got me from going 3TB(PigStorage) to 60 gb. This is insane. I didn't notice any performance improvement though. But I am happy with savings in storage. Thanks! 🙂

Re: Faster and Better Optimized Storage format in Pig?

aervits — Fri, 10 Mar 2017 00:36:11 GMT

@Adnan Alvee that is impressive indeed, ORC has additional benefits you will see on the Hive side. Glad you found it of use.