question Re: Pig ParquetStorer is not working in Archives of Support Questions (Read Only)

Pig ParquetStorer is not working

IbrahimJ — Fri, 15 Jan 2016 01:41:16 GMT

Hi There,

We are getting the following error when using ParquetStorer in Pig

ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Cannot instantiate class org.apache.pig.builtin.ParquetStorer (parquet.pig.ParquetStorer)

We are using HDP-2.3.4.0-3485 version.

Appreciate if any one have any pointers on this.

Thank you,

Ibrahim

Re: Pig ParquetStorer is not working

aervits — Fri, 15 Jan 2016 02:13:50 GMT

you need to download the parquet jar, upload it to the cluster and register the Parquet jar. HDP doesn't ship with Parquet out of the box. @Ibrahim Jarrar

here's an example.

Re: Pig ParquetStorer is not working

GeeKay2015 — Fri, 15 Jan 2016 02:45:01 GMT

--Register the jars

--store in parquet format

SET parquet.compression gzip or SNAPPY;

STORE table INTO '/path/to/table' USING parquet.pig.ParquetStorer;

-- options you might want to fiddle with

SET parquet.page.size 1048576 -- default. this is your min read/write unit.

SET parquet.block.size 134217728 -- default. your memory budget for buffering data

SET parquet.compression lzo -- or you can use none, gzip, snappy

STORE mydata into '/some/path' USING parquet.pig.ParquetStorer; --Reading mydata = LOAD '/some/path' USING parquet.pig.ParquetLoader AS (x: int, y int);

Re: Pig ParquetStorer is not working

aervits — Wed, 03 Feb 2016 09:50:40 GMT

@Ibrahim Jarrar has this been resolved? Can you provide your solution or accept the best answer?

Re: Pig ParquetStorer is not working

aervits — Sat, 26 Aug 2017 00:07:47 GMT

Here's a much cleaner working example tested with HDP 2.6

wget http://central.maven.org/maven2/org/apache/parquet/parquet-pig-bundle/1.8.1/parquet-pig-bundle-1.8.1.jar
hdfs dfs -put
parquet-pig-bundle-1.8.1.jar .
pig –x tez
REGISTER
hdfs://dlm3ha/user/centos/parquet-pig-bundle-1.8.1.jar;
// words is a CSV file with five fields
data = load 'words' using
PigStorage(',') as
(f1:chararray,f2:chararray,f3:chararray,f4:chararray,f5:chararray);
store data into
'hdfs://dlm3ha/user/centos/output' using org.apache.parquet.pig.ParquetStorer;

Re: Pig ParquetStorer is not working

rene_lobera1 — Tue, 30 Jul 2019 07:32:04 GMT

i used:

register parquet-pig-1.10.1.jar;
register parquet-encoding-1.8.2.jar;
register parquet-column-1.8.2.jar;
register parquet-common-1.8.2.jar;
register parquet-hadoop-1.8.2.jar;
register parquet-format-2.3.1.jar;

base = LOAD '/XXX/yyy/archivo.parquet' USING org.apache.parquet.pig.ParquetLoader AS (

xxx:chararray,
yyyy:chararray,

...

)
;

and ok