question Re: Pig - Load data from two different path in Archives of Support Questions (Read Only)

Pig - Load data from two different path

SatishS — Sat, 27 May 2017 09:23:29 GMT

Hi Friends,

I have question on Pig script. I have to load data from two different HDFS paths into single Pig relation.

Ex: /data/input1.csv and another file is in /inputdata/input1 or input2.csv.

Is it possible to load these two tables in to single Pig relation?

Thanks,

Satish.

Re: Pig - Load data from two different path

gkeys — Sat, 27 May 2017 11:02:48 GMT

If these two sources have the same schema it is a simple manner of using the UNION operator to do these three steps:

Source_1 = LOAD "/data/input1.csv" USING PigStorage(',') ...
Source_2 = LOAD "/data/input2.csv" USING PigStorage(',') ...
Source = UNION Source_1, Source_2;

See these references for elaboration:

Re: Pig - Load data from two different path

SatishS — Sat, 27 May 2017 16:09:11 GMT

Thanks Greg, but is there anyway to load both files from different path into single relation using LOAD?

Re: Pig - Load data from two different path

gkeys — Sat, 27 May 2017 18:31:54 GMT

@Satish Sarapuri Yes, you can GLOB the filename pattern. This will work work:

Source = LOAD '/data/input{1,2}.csv' USING PigStorage(,)...

You can use other GLOB patterns. See https://books.google.com/books?id=Nff49D7vnJcC&pg=PA60&lpg=PA60&dq=hdfs+glob&source=bl&ots=IjkvXt9zUn&sig=AKjzNQ77C9BaRgZyqvkJ4YFI7gU&hl=en&sa=X&ved=0ahUKEwirt5_O_I_UAhUE1CYKHTtCDqIQ6AEITzAH#v=onepage&q=hdfs%20glob&f=false

Re: Pig - Load data from two different path

SatishS — Sat, 27 May 2017 21:50:31 GMT

@Grey Keys, both source data is in different paths.

Re: Pig - Load data from two different path

gkeys — Tue, 30 May 2017 19:54:40 GMT

@Satish Sarapuri You can use globs anywhere in the path (not just the filename). There are quite many operators for globs (similar to linux) as shown in the above link, so if there is enough in common with the paths you should be able to leverage globs for the differing parts. If none of that works, you could still use the globs with full paths:

Source = LOAD '/{path1,path2}' USING PigStorage(,)...
where path1 and path2 can be any file path.

Re: Pig - Load data from two different path

SatishS — Mon, 05 Jun 2017 05:15:05 GMT

@Greg Keys

Both source files are in different paths.

Re: Pig - Load data from two different path

broute — Thu, 11 Apr 2019 01:24:42 GMT

Hi, I am a new one of Big Data. This code is like Union? So mean, @Greg Keys you write two codes. They are working same? Thank you for answering...