@Santanu Ghosh, You have to use PARALLEL with any operator that starts a reduce phase like GROUP, JOIN, CROSS, DISTINCT etc. I have mentioned usage of PARALLEL with an example data set 1) Put the data.csv into hdfs [qa@vnode-68 root]$ hdfs dfs -put data.csv /user/qa/ . 2) Check the content of the data file [qa@vnode-68 root]$ hdfs dfs -cat /user/qa/data.csv
abhi,34,brown,5
john,35,green,6
amy,30,brown,6
Steve,38,blue,6
Brett,35,brown,6
Andy,34,brown,6
. 3) Run the pig script which will group users by color and dump the output into hdfs [qa@vnode-68 root]$ pig
grunt> data = LOAD '/user/qa/data.csv' using PigStorage(',') as (name:chararray,age:int, color:chararray,height:int);
grunt> b = group data by color parallel 3;
grunt> store b into '/user/qa/new' using PigStorage(',');
grunt> quit . 4) Check the output folder to make sure that 3 files are created [qa@vnode-68 root]$ hdfs dfs -ls /user/qa/new
Found 4 items
-rw-r--r-- 3 qa hdfs 0 2018-03-17 16:28 /user/qa/new/_SUCCESS
-rw-r--r-- 3 qa hdfs 0 2018-03-17 16:28 /user/qa/new/part-r-00000
-rw-r--r-- 3 qa hdfs 80 2018-03-17 16:28 /user/qa/new/part-r-00001
-rw-r--r-- 3 qa hdfs 51 2018-03-17 16:28 /user/qa/new/part-r-00002 Additional reference : https://pig.apache.org/docs/r0.15.0/perf.html#parallel If this helps you, please click on the Accept button to accept the answer. This will be really useful for other community users. . -Aditya
... View more
